RESEARCH ARTICLE Open Access

How to Design a Remote PatientMonitoring System? A French Case StudyMarie Ferrua1* , Etienne Minvielle1,2, Aude Fourcade1, Benoît Lalloué3, Claude Sicotte1,4, Mario Di Palma1,5

and Olivier Mir1,6

Abstract

Background: Remote Patient Monitoring Systems (RPMS) based on e-health, Nurse Navigators (NNs) and patientengagement can improve patient follow-up and have a positive impact on quality of care (by limiting adverseevents) and costs (by reducing readmissions). However, the extent of this impact depends on effectiveimplementation which is often restricted. This is partly due to the lack of attention paid to the RPMS design phaseprior to implementation. The content of the RPMS can be carefully designed at this stage and various obstaclesanticipated. Our aim was to report on an RPMS design case to provide insights into the methodology required inorder to manage this phase.

Methods: This study was carried out at Gustave Roussy, a comprehensive cancer centre, in France. Amultidisciplinary team coordinated the CAPRI RPMS design process (2013–2015) that later produced positiveoutcomes. Data were collected during eight studies conducted according to the Medical Research Council (MRC)framework. This project was approved by the French National Data Protection Authorities.

Results: Based on the study results, the multidisciplinary team defined strategies for resolving obstacles prior to theimplementation of CAPRI. Consequently, the final CAPRI design includes a web app with two interfaces (patientand health care professionals) and two NNs. The NNs provide regular follow-up via telephone or email to managepatients’ symptoms and toxicity, treatment compliance and care packages. Patients contact the NNs via a securemessaging system. Eighty clinical decision support tools enable NNs to prioritise and decide on the course ofaction to be taken.

Conclusion: In our experience, the RPMS design process and, more generally, that of any complex interventionprogramme, is an important phase that requires a sound methodological basis. This study is also consistent withthe notion that an RPMS is more than a technological innovation. This is indeed an organizational innovation, andprinciples identified during the design phase can help in the effective use of a RPMS (e.g. locating NNs if possiblewithin the care organization; recruiting NNs with clinical and managerial skills; defining algorithms for clinicaldecision support tools for assessment, but also for patient decision and orientation).

Keywords: Implementation, Remote Patient Monitoring system, Care coordination, Complex intervention, Oncology

© The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you giveappropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate ifchanges were made. The images or other third party material in this article are included in the article's Creative Commonslicence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commonslicence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtainpermission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to thedata made available in this article, unless otherwise stated in a credit line to the data.

* Correspondence: marie.ferrua@gustaveroussy.fr1Capri program, Research Division, Gustave Roussy, Villejuif, FranceFull list of author information is available at the end of the article

Ferrua et al. BMC Health Services Research (2020) 20:434 https://doi.org/10.1186/s12913-020-05293-4

BackgroundThe delivery of healthcare services for patients withchronic diseases requires more effective coordination be-tween professionals and patients/relatives along the carepathway. In response, many health care organisationshave implemented Remote Patient Monitoring Systems(RPMS). This type of intervention programme can im-prove patient follow-up and have a positive impact onthe quality of care (reducing toxic effects, improvingtreatment compliance, limiting adverse events) andhealth-related costs (reducing the duplication of pre-scriptions and hospital readmissions) [1–5].According to the Chronic Care Model [6, 7], an RPMS

comprises three distinct components that facilitate thecoordination of patient care: a) organisational methods(e.g. patient navigation program) [8, 9], b) e-health tech-nology (e.g. web portal, apps) [10, 11] and c) patient en-gagement through information and training (e.g. health-literacy tools) [12, 13]. By combining one or more ofthese key components, many RPMS have been appliedaround the world in a variety of chronic diseases in re-cent years with some spectacular developments [14–16].However, two recent literature reviews reported mixed

results in terms of the effective impact of RPMS onquality of care and cost management [17, 18]. This ispartly attributed to the fact that effective RPMS imple-mentation remains a significant challenge. It depends onthe processes accepted in relation to professionals andpatients as well as on the local organizational context. Itis important to understand the factors that can influencepatient behaviour and healthcare practices, as well as thespecific features of the intervention programme per seand its implementation context [5, 17, 19, 20]. The func-tional components of the RMPS must also be clearly de-scribed to facilitate implementation and replication.Hoffmann et al., for example, reported that only 39% ofnon-pharmacological interventions are adequately de-scribed. This lack of precision leads to replication diffi-culties in other settings [21]. Finally, it is obvious thatthe evaluation protocol must be designed ex ante to theintervention [22]. Otherwise, some data required may beomitted. Overall, these observations indicate a carefulapproach should be adopted when designing any RPMS.The more consideration given to the afore-mentioned is-sues during this design phase, the more effective the im-plementation is likely to be [20]. A precise design canreduce the risk of ineffective implementation, avoid add-itional re-design costs and prevent replication problems[19, 23].In order to investigate the design process, an RPMS is

defined as a complex intervention involving several in-teractions between various individuals, organisations andtools which in this case refer to the combination ofstages outlined above [19, 24]. An in-depth literature

review has been proposed to guide the development, im-plementation and evaluation of such complex interven-tions [19, 25–28]. One major reference is the UKMedical Research Council (MRC) that defines fourstages comprising key functions and activities: Develop-ment, Feasibility/piloting, Evaluation, and Implementa-tion [25, 29]. This widely used guideline is of particularinterest for interventions involving e-Health technologiessuch as RPMS [30]. It helped us to use the interventiondesign process as the core development phase of thisframework.The aim of this paper is to report on our RPMS

process design experience based on the CAPRI (Cancér-ologie Parcours Région Ile de France) Case Study. TheRPMS, recognised as such by the French public author-ities, consists of 2 NNs and technological support (atelephone platform and a web application). Its added-value has been assessed through a Randomized ClinicalTrial (RCT) that significantly demonstrated an increasedfollow-up of the medical prescription, a reduction in theeffects of severe toxicity, a decrease in hospital days, anda better patient experience in the follow-up of their dis-ease [31]. The RPMS is the result of a design processcarried out to improve the management of cancer pa-tients throughout the care pathway. Cancer is an idealfield for developing RPMS since it is a major chronicdisease (leading cause of death in addition to having sig-nificant social and economic consequences) [32, 33], andrequires organizational innovations for improving coord-ination along the care pathway. We based our design re-search protocol on the MRC (Medical Research Council)framework [25, 29]. A series of research studies havebeen conducted in accordance with the MRC’s guide-lines to identify the evidence base, develop the appropri-ate theory and to model process and outcomes duringthe development phase. The contributions are two-fold.Firstly, they provide insight into an oncology-specificRPMS and its design process, which is a current issue.Secondly, it contributes to the methodological debate re-garding guidelines for developing complex interventions.

MethodsThe implementation and project management teamThe RPMS under test, namely, CAPRI (CAncerologieParcours Région Ile de France), was trialled at the Gus-tave Roussy Institute (Villejuif, France), a leading Euro-pean cancer centre with 449 beds and 94 day-careplaces, treating over 48,000 patients (year 2018).The purpose of CAPRI was to develop a RPMS dedi-

cated to patients receiving oral cancer medication - atreatment that has been used more extensively in recentyears [34]. The use of such treatments is associated withorganising and coordinating challenges [34–36]. Al-though most patients prefer oral therapy to IV therapy,

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 2 of 16

they assume greater responsibility with oral therapies,and may encounter difficulties such as toxicity, whichcould affect treatment compliance. Furthermore, visitsto health facilities are less frequent with primary careprofessionals potentially being the primary point ofcontact.The initial stage of the project was devoted to the de-

sign of CAPRI (mid-2013-2015), and the second stage toimplementation and evaluation (2016–2019). Threegroups were formed depending on the work being car-ried out and the individual areas:

1) An expert group focused on the scientific aspects.Several disciplines were represented: oncologists,pharmacists, and health service researchers(management science and biostatistics);

2) A functional group was set up comprising hospitalmanagement personnel from various departments:Quality, Medical Information, Information SystemManagement, Nursing Directorate, Outpatients,Pharmacy and Finance to make administrativedecisions;

3) An operational working group including researchersfrom the expert group, and staff from differentclinical departments to plan the daily agenda.

During the initial intervention design stage, the expertgroup met monthly, the functional group once every 3months, and the last group on a weekly basis dependingon project progression, in addition to telephone callsand emails.The Principal Investigator of the evaluation

programme was the Outpatient Senior Oncologist givenhis interaction with other healthcare professionals (MdPfollowed by OM). The Scientific Lead was a senior re-searcher in Healthcare Management Science (EM). AProgramme Coordinator was also appointed to overseeall project management issues (MF).

Methodological principles to support the RPMS designprocessThe design process was managed by the multidisciplin-ary team comprising the three groups. Their role wasbacked by various studies carried out at Gustave Roussy.Eight studies were finally outlined, based on the threeprinciples covered in the MRC framework during thedevelopment phase [25]: (1) Identify the evidence base:to identify existing knowledge about similar interven-tions and the methods used to evaluate them, (2) Iden-tify/develop an appropriate theory to promote thetheoretical understanding of the likely process of changeby drawing on existing evidence and theory and (3)Model process and outcomes prior to full-scaleevaluation.

The CNIL approved each study. As a result, severalstages in the design process were carried out between2013 and 2015 (see Fig. 1).The various studies carried out in support of the

CAPRI design process are also summarised in Table 1.Some studies have been published in more detail else-where [37–41]. The four exploratory studies (studies 1to 4) provided a preliminary draft of the CAPRI designincluding selected components (health technologies andnew organisational methods involving nurses specialisingin coordination) and the main functions and interaction.The intervention proposal was then presented to poten-tial stakeholders (study 5) to model the process and out-comes. A patient study was conducted simultaneously toidentify the unmet information needs of cancer patientsand understand the reasons behind patient dissatisfac-tion (study 6). A quantitative study on the appropriate-ness and potential avoidance of emergency visits wasalso carried out to assess the potential contribution ofthe intervention (study 7). Finally, interviews were heldwith Gustave Roussy professionals involved in patientfollow-up (oncologists, support teams) to model patientmonitoring and develop the necessary monitoring tools(protocols, clinical decision support tools) (study 8).

ResultsThe results are divided into two parts: the first part givesan insight into the RPMS design process whilst the sec-ond part describes the final Capri design.

The RPMS design processThe main results of the studies conducted during thisdesign phase are summarised in Table 2.They contributed in several stages to the design of the

intervention programme and the assessment methodsemployed.

RPMS contentThe literature reviews (studies 1 and 2, Table 2) takinginto consideration the experiments carried out to coordin-ate care reported that a very wide range of tools and orga-nisations are used to improve the care pathway. Someinterventions are based on apps or tablets whereas othersuse telephone platforms. The role of case manager, coord-ination nurse or senior nursing clinician is often perceivedas value added (studies 2 and 4, Table 2). As a result, thetwo main components retained, e-health technologies andnew organizational methods with the implementation ofNurse Navigators (NN), were deemed potentially appropri-ate to meet coordination needs identified in the specificcase of CAPRI. Patient commitment, another importantfactor in the successful patient-professional relationship, islinked as much with IT usage as with the quality of theNNs relationship (studies 3 and 6, Table 2).

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 3 of 16

Each CAPRI component was also analysed in order topredict implantation problems. The introduction of ITwith a patient survey (median age 53.4 years) [38], hasallowed the most appropriate technological tool (weband mobile app with two interfaces: patients and out-patient professionals) to be identified alongside the pri-ority features to be developed to make the technologicaltool attractive for patients (studies 3 and 5, Table 2).Similarly, the combined study on home care coordin-ation activities has prompted a better understanding ofthe current practices and skills required for the NNs role(study 4, Table 2) [39]. Investigations have shown thatNNs have to deal far more with organisational issues orroutine activities in the life of a patient (e.g. how totravel with oral therapy) than with clinical ones.Under the supervision of the three groups, these ex-

ploratory studies have led to a precise design, suggestinga technological tool, including a definition of these func-tions and a detailed description of the NNs role. The ini-tial design could be discussed with potential users inorder to model process and outcomes and define the

major expected outcomes and the most relevant targetpopulation for the design of CAPRI.

Clinical decision supportThe interviews carried out during study 5 (Table 2)highlighted the fact that NNs have clinical decision sup-port tools to standardise patient follow-up in the veryleast (initial consultation, follow-up frequency andmethods, evaluation parameters) to assess the situation,and finally to define the action to be taken. These toolswere devised and validated with referral oncologists andhealth care support staff (study 8, Table 2), they weredevised on the basis of NCI-CTCAE-V4 [45]. The ques-tions to be put to the patients were defined in order todetermine the severity of the recurring events, action(procedure to implement) and results (assessment of ac-tions). These algorithms led to 3 action-based principlesfor NNs: advice given to patients, the organisation of aconsultation (hospital or GP), and the organisation of ahospital admission. Ultimately, 80 clinical decision sup-port tools have been developed allowing NNs to

Fig. 1 CAPRI design: overview of supporting research

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 4 of 16

Table 1 Study methodology for designing CAPRI

N° Studies Objectives Study design Methods Period

1 Literaturereview [37]

Select existing literature reviewsto identify interventions toimprove coordination on thecancer pathway

Literature review analysis Selection of literature reviews viaPubMed and extraction of interventionswith a demonstrated effectiveness

2013

2 IT usages in cancercare coordination

- IT state-of-the-art in cancer care- Identify good practices tosupport efficientimplementation

Literature review Literature Review (pubMed andCochrane Library)- Extraction of IT uses reported in theliterature and classification

- Recommended formulations

2013/2014

3 Patient use ofinternet-basedtechnologies [38]

- Understand the current level ofusage of internet-basedtechnologies by patients

- Assess their intention touse them for their health

Quantitative survey GustaveRoussy’s outpatientdescriptivestatistics and correlationanalysis

Questionnaire-based survey carried outwithin seven outpatient departmentsover 7 days.- 3-part Questionnaire:(i) Use of internet through computers,mobile phones and tablets(ii) Willingness to use informationtechnologies for health purposes(iii) Socio-demographics

2013/2014

4 Home carecoordinationactivities andskills [39]

- Identify the need categoriesof patients and primary careproviders for home carecoordination

- Quantify the volume of theactivity generated by eachcategory of needs.

Qualitative and quantitativeanalysis

Mixed method:(i) Qualitative phase: interviews withpatients and focus groups with the NNsof the Coordinating Outpatient Care(COC) department at Gustave Roussy(ii) Quantitative phase: phone calls(made by both patients and primarycare providers) received at the COCdepartment. The caller, reason for thecall and procedure performed weresystematically reported and thenanalysed.

2014

5 Care coordinationneeds

- Understand operational careprocesses

- Identify care coordinationneeds

- Define how technological toolsand nurses could preventdifficulties and facilitate carecoordination between patientsand professionals

- Choose the location of NN

Interview survey Interview survey with patients, hospitalpractitioners, primary healthcareproviders and other hospitalprofessionals

2014

6 Patient informationneeds [40]

Identify the unmet informationneeds of cancer patients andunderstand the reasons behindpatient dissatisfaction

Interview survey andshadowing

Interviews with cancer patientsattending a Meeting and InformationArea (ERI) at Gustave Roussy and focusgroups with ERI professionals.Data were analysed using vertical andhorizontal open coding

2015

7 EmergencyDepartmentreferrals [41]

Describe and quantify theappropriateness and potentialavoidance of EmergencyDepartment referrals

Electronic medicalrecord review

Prospective review of the electronicmedical charts of patients admitted insuccession to the EmergencyDepartment in August 2015.The appropriateness of referrals wasassessed using a nationally validatedclassification system and local criteria.Potentially avoidable referrals wereassessed using international classificationsystems and local criteria

2015

8 Clinical monitoringneeds

- Identify clinical monitoringparameters

- Define the monitoringguidelines

Interview survey - NNs- Oncologists (referral physicianaccording to site and supportive care)

2014/2015

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 5 of 16

Table 2 Key findings from combined CAPRI design studies

N° Studies Sample Main results Principal findings fordesign intervention andimplementation

Principal findings for evaluation

1 Literature review 3 literature reviewidentified[42–44]

Effective intervention based onliterature reviews:- Patient information- Decision –making aidsfor patients

- Audiotaped consultation- Follow-up by nurses- Follow-up by GPs- Case management- One-stop clinic- Shared-care programme

Selection of componentsfrom the CCM

Combination of randomisedcontrolled trial (RCT) andprocess evaluation

2 IT usages incancer care

coordination

46 articles analysedin realistic literaturereviews

Identification of six uses of TIC:document management,dissemination of informationand pooling of patient data,communication betweenstakeholders, aid in clinicaldecision-making, patienteducation and level ofindependence, personalisationand coordination of carepathway

- Definition of the functionalbasis provided by thetechnological tools usedin the interventionprogramme

- Implementationrecommendation: promotethe sharing of informationand system integration,rigorously plan the designof the intervention, improveproject management, workon tool ergonomics, planthe secure data strategy

Need to devise a robustevaluation strategy to assessthe quality of life, satisfaction,organisational and economicimpacts as well as the clinicaloutcome.

3 Patient use of IT n = 1371questionnaires(participationlevel = 85%)Median age ofpatients: 53.4 years,70% were females

Access and use:93% had home access to theInternet, 71% used a mobilephone every day and mostpatients reported never usingtabletsWillingness to use IT for theirhealth:The most useful features:- Having access to electronicrecords, completion of a self-test to assess health status,communicating with physicianvia email, bookingappointments and obtaininginformation about their disease

- The least useful features:chatting with peer patients,communication via video

Perceived ease of use:84% confirmed that they wereable to use a computer, tablet orsmartphone

- Study provided cancerpatients with anopportunity to use IT forhealth purposes, no majorobstacles identified but theeffects of age andsocioeconomic statushave to be addressed.

- No need to equip thepatient with any additionalmaterial (e.g. digital tablet).

- Selection of priorityfunctions to be integratedin the tool (data collectionsystem, secure messagingsystem, information sourceprovided, etc.).

- Data security requirements- Key contacts

To be considered in processevaluation:Acceptance of the IT tool, patientprofile with regard to IT,frequency of use and changes inuse over time

4 Home carecoordinationactivities andskills

Qualitative phase:17 interviews withpatients and 2 focusgroups with NNsQuantitative phase:543 phone callsreceived via COCplatform

Five categories of NNs-relatedactivities defined as:1. Patient monitoring (e.g.:reporting side effects)2. Navigation assistance (clinicalpathways)3. Managing technical problems(difficulties in drug ormedical device delivery orequipment malfunction)4. Explaining care protocols (e.g.clarification about the applicationof a drug prescription)5. Collecting and transmittingpatient dataAlthough a significant proportionof the NNs’ activities involve

- Definition of the NNs profileand development of thejob description (role of casemanager with clinical skills,knowledge of outpatientcare and the healthcaresystem)

- Need to develop tools fornurses to assist in themanagement of patientfollow-up (clinical decisionsupport, protocol)

To be considered in processevaluation:- organisational change triggered- Characteristics of NNs activities

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 6 of 16

Table 2 Key findings from combined CAPRI design studies (Continued)

N° Studies Sample Main results Principal findings fordesign intervention andimplementation

Principal findings for evaluation

patient monitoring (29%), mostof the requirements (71%) relateto organisational issues.

5 Carecoordinationneeds

59 individuals met, 45interviews conducted−19 Patients−11 Communityprofessionals(GP,Private nurses,Dietician, Pharmacist)- 29 Gustave RoussyProfessionals

Potential benefits of the digitaltool- Standardisation of follow-upinformationProvision ofpractical information ona daily basis

- Functions: long-distanceconsultation, documentdispatch (results, evaluation,photos), monitoring of vitalparameters, organisation ofappointments, storage andpermanent access toinformation, list ofpersonal contacts

Potential benefits of NNs- To answer telephone callsand receive alerts

- To have explanatoryconsultations in addition tothe normal reporting system(diagnosis, relapse,discontinuation oftreatments, etc.)

- To support patients alongthe pathway

- To send information to variousprofessionals involved in thepatient’s treatment pathway(hospital and community) andto guarantee the link betweenpatient, treating physician andreferral oncologist

NNs location:- Community professionals mayhave a lack of information andtraining in oncology, they havedifficulties in having oncologistexpertise

Warnings relating to thedigital tool:- Does not replace direct ortelephone contact

- The information collected isnot sufficient to trigger adecision and action

- Avoids the risk of intrusionin the patient’s home

Warnings regarding the roleof the NNs – what the NNmust not do:- Manage appointments,guarantee regulations andrefer to emergency unit,responding to medicalalerts and take decisions

Conditions for a successfuloutcome:- Have a baseline (withclinical decision supporttool) which is validated byall the committees, withwarning thresholds andprocedures to follow

NNs’ profile:- Case manager role withclinical competencies,knowledge of theoutpatient and care system

- Ability to interact with theHospital Information System

NNs location:- Hospital: to have an easyaccess to the oncologist’sexpertise

Population selection: patientstreated with oral anticancerdrugsRCT:- Choice of primary evaluationcriterion/endpoint:

Efficacy hypothesis: thanks tofaster management of treatment-related side effects, patientsparticipating in theCAPRI intervention programmewill demonstrate a significantincrease in Relative DoseIntensity (RDI)- Choice of secondary criteria:patient compliance, quality oflife, patient experience, tumourresponse, Progression FreeSurvival, Overall Survival, toxicside effects and economicevaluation (medical andnon-medical costs)Process evaluation:Study of changes inorganisational transformationsand, in particular, the impact ofthe intervention programme onthe oncologists’ workload

6 Patientinformationneeds

19 interviews withpatients

- Patients were looking fortreatment documentation ontreatments but three types ofnon-medical information werealso identified:

a) Information on the carepathway, hospital and on healthcare system in general (e.g.administrativerules, departmental structure);b) Information on supportivecare (e.g. services, activities) andhow to contact professionalsinternally (within the hospital)and externally (e.g. dietician,psychologist);c) Information on living withcancer and its impact on dailyactivities.- Patient dissatisfaction is linkednot only to the lack of medicalinformation but also reflectsother needs, which are not

- Information must beconsidered using anintegrated and holisticapproach to facilitate thepatient navigation processand improve health-relatedliteracy

- Training of healthcareprofessionals is crucial, butthis is not enough. Theintroduction of other, non-carer professionals isnecessary to address awide range of patient-related needs in a more effective and cost-efficientmanner.

Assessment criteria usedin the longitudinal analysis:Acceptance by patients

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 7 of 16

prioritise and define the action to be taken based onalert parameters. This study also highlighted that NNsshould be located within the hospital where patients aretreated in order to have easy access to the oncologist’sexpertise if necessary.

Evaluation protocolAlongside the intervention design, one major point wasto define the entire assessment system prior to the im-plementation process in order to outline the evaluationcriteria, the data to be collected and the targetpopulation.

According to the literature [20, 46], the group of ex-perts also adopted the concept of an evaluation process,which led to the definition of two evaluation methodsfor the design of the protocol to measure the impact ofCAPRI [47]. Initially, as this is a type of study with astrong evidence base, a decision was taken to assess the« clinical » impact of the programme based on a rando-mised, controlled trial with a 700-patient cohort. Thecriteria for this study were determined using preliminarystudies (studies 2 to 8, Table 2). The main evaluationcriterion was defined as the relative dose intensity to as-sess compliance between the dose taken by the patient

Table 2 Key findings from combined CAPRI design studies (Continued)

N° Studies Sample Main results Principal findings fordesign intervention andimplementation

Principal findings for evaluation

taken into account (e.g.expanding on information tomake it understandable anduseful).

7 EmergencyDepartment (ED)referrals

Electronic medicalrecord review:500 referrals relatedto 423 patients

- Referrals were appropriate in61% of cases

- Referrals were deemedpotentially avoidable in 33.4%of cases, potentially avoidablein 14.4% and unavoidable in52% of cases

Opportunities to avoid referralsafter index hospitalisationinvolved this hospital stay ordischarge process in 66 cases(28%), the follow-up period in 59cases (25%), or both in 66cases (28%).Causes of potentially avoidablereferrals may be linked to threemain problems:- A lack of effective care duringfollow-up (lack of medicalexpertise, either on the part ofthe oncologist regardingchronic or intercurrentconditions or on the partof the GP about cancer)

- Care coordination (lack ofinformation for outpatientproviders on referrals, andoutpatient referrals omitted)

- Patient management duringthe index hospitalisation(premature discharge orinadequate assessment ofpost- discharge risk)

- Lack of information frominpatient to outpatientproviders but also vice-versa

- Most inappropriate referralsneeded consultations andnot in a hospital setting

- Merits of the GP to be incontact with the oncologistto improve the relevance ofreferrals

- Need for tools to facilitatecommunication, legalframework development,financial incentives,training in sharedmedical managementand patient education

Criteria regarding readmissionand ED visits were added toprotocol evaluation

8 Clinicalmonitoringneeds

22 interviews withoncologists, NNs, andsupport packages

Drafting of clinical decisionsupport tool in conjunction witha follow-up protocol throughjoint work between the NNs andthe various Gustave RoussyMedical Discipline Leadsregarding the informationprovided by analysing themedical and paramedicalliterature and obtainingexpert opinions.

- Modelling of the follow-upprocess (initial NNsconsultation, frequencyof follow-up, items to beassessed, pooling ofinformation)

- Devising NNs follow-uptools (clinical decisionsupport)

- 80 validated clinicaldecision support tool

Design of NNs activities forimproving evaluation criteria

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 8 of 16

and the one scheduled in the protocol. Secondary end-points were patient compliance with oral anticancertherapy, quality of life, patient experience, tumour re-sponse, Progression Free Survival (PFS), Overall Survival(OS) and the toxic side effects of treatment (severity andquantity). The RCT also includes an economic evalu-ation which adopts a societal perspective, assessingintervention, medical and non-medical costs. Finally,one endpoint focuses on the reason for emergency refer-rals. The results obtained in the Emergency Departmentstudy [41] have reinforced the value of the interventionprogramme in terms of reducing the number of un-necessary visits to the emergency department. Inappro-priate and avoidable visits appear to be caused byinadequate referral to the most appropriate health pro-fessional, which is a key point in the follow-up process.In addition, the difficulty in quantitatively measuring

some of its effects and the behaviour of professionals incoordinating their actions, in particular, resulted in alongitudinal study. Collectively, and based on the ana-lyses of articles in an attempt to RPMS evidence (studies1, 2 and 8, Table 2), this approach was adopted to evalu-ate changes in the use and suitability of both tool andprogramme over time, focusing in addition on the be-haviour of patients and professionals and how changeswere implemented and knowledge was acquired duringthe implementation of the CAPRI programme.

Legal issuesIn addition, regulatory strategies specific to developingthe evaluation of telemedicine/remote medicine systemsin an experimental setting were required. Since theCAPRI follow-up system is based on a telemedicine ac-tivity (remote follow-up), a telemedicine contract had tobe signed with the Agence Régionale de Santé Ile deFrance (ARSIF) (Parisian Regional Health Agency). Thisstep took 7 months and the contract with the ARS Ile deFrance was signed in October 2015. A mandatory pro-cedure was also required with regard to the CommissionNationale de l’Informatique et des Libertés (CNIL) toauthorise the pooling of personal health-related informa-tion. The CNIL procedure was initiated only after signa-ture of the ARS Ile contract, and was obtained in May2016. These legal issues have governed the regulatoryframework in which the CAPRI system can be used.Apart from the timescales imposed by these regulatory

authorities, compliance also impacted system design. Forinstance, the plan was to allow community professionals,and General Practitioners in particular, to documentmutual patient information using the shared patientfollow-up record. This option was abandoned since anagreement would have been required with each healthcare professional in accordance with regulatory

requirements. This system was deemed to be too com-plex for routine use.

A dynamic, iterative processAll the issues outlined above refer to a dynamic designprocess. As summarised in Fig. 2, our design processallowed a simultaneous process to be carried out usingan iterative approach towards the three MRC principleswe applied.This iterative approach allowed various obstacles to be

identified (e.g. real and priority needs, local context spe-cifics) and corrected prior to implementation (e.g. targetintervention to the patients most likely to benefit; cor-rect combination of NNs activities and e-health technol-ogy functions; holistic process design and outcomeevaluation measure).

Capri intervention specificationsThe final CAPRI design includes a web/mobile app withtwo interfaces (patient and professional) and two NurseNavigators (Fig. 3).The organisational aspects are quite important. We

have already described the different clinical decisionsupport tools developed to assist NNs activities. The twoNNs provide regular telephone follow-up to manage pa-tients’ symptoms and toxicity issues, treatment compli-ance and supportive care needs. Patients have access tothe app to record/track data, contact the NNs via a se-cure messaging system, view therapy and side effect in-formation or store documents. The NNs are linked tohealth professionals involved in patient management.

Organisation of NNs activitiesNNs conduct an initial assessment interview with eachpatient in-person or over the phone to identify his orher needs. The patient interview also includes a reviewof treatment, medical prescriptions and appointments.NNs then prepare the individual patient electronic med-ical record on the CAPRI application. Following this ini-tial phase, NNs ensure patient follow-up (e.g.temperature, weight, pain, diet) remotely, through tele-phone interviews and emails, from Monday to Friday,during office hours only (from 9 am to 5 pm). Patientsbenefit from a regular phone follow-up in addition to in-dividual contact depending on access difficulties, needs,and resources. In addition, NNs help patients to identifyand overcome obstacles, provide health and practical in-formation as well as emotional support, help patients toorganise their appointments, help them to understandtheir conditions and treatments and help them to be ac-tively involved in their care. They also forge links be-tween the patient and hospital professionals and primarycare providers (GP, private nurse, pharmacist, etc.) whoare given access to the CAPRI application with the

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 9 of 16

Fig. 3 Final CAPRI design

Fig. 2 CAPRI design: an iterative process

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 10 of 16

patient’s consent. To this end, NNs ensure that consult-ation reports, examination and test results and newmedical prescriptions are available on the CAPRI appli-cation to all authorised healthcare providers at the timeof an appointment. NNs inform health care providersabout patient appointments, new treatments, new symp-toms or difficulties, as required. Patient monitoring isdescribed in a specific protocol prepared and validatedby the expert group. During the initial assessment, eachpatient is given a starter box, which includes the follow-ing: login data to gain access to the portal, instructionsfor use and covering letters for healthcare providers, re-quired (with information on the web portal, instructionson how to create an account and the, NNs’ contactdetails).

Web/mobile CAPRI applicationThe CAPRI application is available in web or mobile ver-sion and provides an interface to connect patients, hos-pital professionals, primary care providers and NNs.The application offers patients several modules as

shown in Table 3.The CAPRI application provides NNs with a dash-

board to enable them to monitor the individual elec-tronic medical records of patients. Each time a patient iscontacted, the NN can create intervention reports torecord what they have done or discussed, and transmitthe information to the professionals previously indicatedby the patients. These professionals can log on the portalto communicate with the NNs online and access therelevant patient information. The system also generatesautomatic alerts which are sent to the patients or theNNs. The alerts and patients’ requests can be generatedin different ways: 1) automatically, via the app, for in-stance while reporting follow-up measures (if the pa-tient’s parameters are below or above predefinedthresholds); 2) by the NNs during regular follow-ups; 3)by messaging/calling the patient or the professionals.The NNs assess the alert grade based on clinical decisionsupport tools and determine the action to be taken ac-cording to navigation algorithms. Depending on thegrade, the NNs can give advice, refer the patient to his

or her primary care physician, or a Gustave Roussy pro-fessional or contact the relevant services in order to ar-range hospitalisation or schedule an appointment for thepatient.

DiscussionThis paper discusses the design of the CAPRI – anRPMS intended to improve the care pathway for cancerpatients receiving oral medication. Two findings cometo light.Firstly, it outlines the crucial role of the design phase

and provides an insight into the method required tocarry out the process.The way in which the system is accepted by patients

and healthcare professionals as well as hospital managersis a key factor in effective implementation that startsduring this design phase [48]. One major criterion wasthe considerable work carried out by the three groupsfocusing on identifying the coordination difficulties andpriority needs in order to grasp a better understandingof the context and to define the main problem to be ad-dressed by the intervention. Particular focus was givento the setting-up of these three groups (expert, func-tional and transversal working groups) and to incorpor-ating individual contributions to provide an overall viewof the intervention programme [49, 50]. The close col-laboration between the three groups and within variousdisciplines and functions has allowed the programme tobe designed in line with the real-life context, thus creat-ing psychological ownership whilst better addressing theneeds of patients, clinicians and managers. These are keyfactors in successful implementation [51, 52].Our experience also shows that evaluation design must

be carefully analysed in advance as a process evaluation.In terms of process and outcome evaluation measures,the work on identifying the target population, outcomesand programme content was crucial to improve the de-sign, criteria and indicators to be followed in the rando-mised control trial, as noted elsewhere [53, 54] and inprocess evaluation. Another important point was to de-fine a combined method comprising a randomised, con-trolled trial and a longitudinal qualitative approach

Table 3 Description of the main modules of the CAPRI patient application

Modules Description

Messages Secured messages to contact NNs

Follow-up Tracking of follow-up measures (e.g. temperature, weight, pain, ingest) and if necessary, patientreporting of other symptoms

Appointments schedule Display and save appointments on a personal schedule

Contact Have access to an address book with contact details of professionals enrolled and other useful numbers

Information Have access to reference websites providing information about the disease, the treatment and their side effects

Storage Download, save and file documents relating to patient care (e.g. clinical and biological exams, patient medical records)

Reminders Schedule reminders to take medications, arrange an appointment, plan exams, document personal measures

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 11 of 16

which requires process evaluation. This process evalu-ation decision was taken in response to two objectives:to understand and describe how the system works inpractice and to assess the level of suitability and retracethe dynamics of the latter in terms of patients and pro-fessionals alike. Indeed, both objectives are required inorder to explain the results observed during the rando-mised study period [20, 46]. Thus it is a case of estab-lishing a link over time between the dynamics of systemsuitability (acceptance method and sustainability/con-tinuity) and changes in the results obtained within therandomised study context. Furthermore, in terms of thelatter, the aim is two-fold. On the one hand, the rando-mised study seeks to highlight the contextual and behav-ioural factors that promote or hamper the implantationand continuity of co-ordination systems (the evaluationof the effects could not be detected to adequate extentby the randomised study and did not provide any justifi-cation, regardless of whether the effects were positive ornegative). On the other hand, the longitudinal studyhighlights the key issues through a scientific approach,driving and optimising the implementation of coordin-ation systems more effectively in line with localrequirements.These insights can contribute to the knowledge of de-

velopment and implementation processes of healthcareintervention programmes. We based our research proto-col on the MRC framework for the development andevaluation of complex interventions [25, 29]. The threeprinciples of the development phase (i.e. “identify theevidence”, “develop an appropriate theory” and “modelprocess and outcomes before a full scale evaluation”),were very helpful. In the experiments reported in thescientific literature, existing evidence in terms of inter-vention content, implementation strategy and processevaluation measures was sparse. Following the MCRprinciples, we collected evidence about the relevant tar-get population, anticipated outcomes and the most suit-able intervention. We developed the underlying theoryand finally designed the intervention programme in op-erational terms, modelling the implementation strategyand the process and outcome evaluation measures. In-deed, the exploratory survey carried out initially and theliterature review on care coordination requirementshighlighted the priorities in terms of both the targetpopulation and outcomes. An intervention programmedesigned to address numerous requirements or severaloutcomes at the same time may produce confused andimprecise results. Similarly, these guidelines allowed theintervention content to be finely tuned following an it-erative approach [25]. Although le MRC Frameworkproved useful in establishing the broad lines of the study,particular attention must be paid to the initial steps con-cerning the key elements in the « Development » stage.

In fact, the method does not state this but, in our opin-ion, it is vital that interactions and repetitions are carriedout between the « Developing appropriate theory », and« Modelling process and outcomes » phases describedduring this stage. The preliminary studies carried outduring the development stage challenged the content ofthe intervention programme on many occasions andallowed us to define the interactions between the variouscomponents to align the evaluation criteria with the finalactions taken within the intervention programme [55].Furthermore, the regulatory and legal constraints seeman inherent part of the intervention design process whenworking on innovation programmes such as RPMS. Interms of our experience, their impact on the length oftime to the implementation authorisation stage was sig-nificant and it seems vital to include this complex issueat the design stage [50].Secondly, our experience shows the importance of

organizational aspects in the RPMS context, preciselythe coordination of care required for the organization ofa remote follow-up of patients, as noticed elsewhere[56]. From a theoretical standpoint, many theories havealready pointed out that the effectiveness of telemedicinecan only be understood in its use, i.e. in social practices[57], or as a technology-in-practice in practice [58, 59].In other words, the coordination of care cannot be pre-dicted, and therefore, there is a limit to consider theseorganizational aspects during the design stage. In thiscontext, this study provides elements on the aspects ofcare coordination that can be reasoned during the designphase. Our study shows that far from a definition ofworking rules of care coordination, some “principles”orienting its implementation, can be identified. The finalCAPRI design provides the following principles that rep-resent guidelines when designing coordination of careaspect into RPMSs:-NNs have already proved their added-value in the pa-

tient management of different chronic diseases [14–16].Here, we highlight that through their actions, NNs canenhance patient engagement in remote dialogue via themobile app or telephone, by tailoring information, ac-cording to their needs [60], trigger clinical alerts, and de-velop coordination with the patient and otherprofessionals (e.g. pharmacists, general practitioners,nurses). It requires a mix of clinical and managerial skills[61] that can lead to profile of NNs required.-The clinical decision support tools provide a basis

for defining the most appropriate answer for specificpatient requirements as well as the most appropriatedirection based on the clinical severity. Designingclinical decision support tools that are not onlyguides for evaluation, but also for orientation, what-ever the organizational conditions to accomplish it,represent another recommendation.

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 12 of 16

-The need to link NNs to the technological applicationin order to process information more efficiently and dir-ect patients accordingly suggests that there is a need tolink the three factors of the Chronic Care Model [6, 7]:use of technological innovation (mobile app), the devel-opment of new coordination roles (NNs) and patientcommitment. Furthermore, this choice prevent physi-cians from being overwhelmed by demands that are notrelated to their clinical expertise, thus allowing them tooptimise their workload [62]. At the same time, it is im-portant for NNs to be able to coordinate easily withmedical oncologists. This has led to NNs located withinthe facility rather than on an external platform to limitthe risk of distance or “physical boundary” [63]. Such achoice is also related to the hospital’s ability to recruitnurses, which depends on the number of patientsfollowed.-Finally, the selection of the feedback support, between

the application allowing automated feedback and thephone, generally preferred by patients but more time-consuming for NNs, is another key point. In our design,we have integrated both options, the use of the RPMSshowing the dominant use of the phone on the applica-tion afterwards [64, 65].This study has various limitations. This was initially a

specific case of RPMS, in a specific setting, namely Gus-tave Roussy, with the aim of monitoring patients receiv-ing oral medication. All of these specific features requireadditional research in other settings, with other objec-tives, to define RPMS models. Secondly, the effective useof the RPMS and the study outcomes are not presentedin the paper. The experimentation gave positive out-comes [31], and showed different insights about the useof the RPMS. As such, they suggest this type of designstrategy can lead to positive outcomes, but it requiresfurther evaluation to understand the relationships be-tween the design phase, the implementation step, andthe outcomes achieved. Thirdly, the strengths of our de-velopment process include the interactive approachcombining the evidence base, theoretical framework andthe involvement of a large network of stakeholders.However, this requires an important research investmentin terms of time and money which is not always feasible.Fourthly, the use of the MRC framework meant thatgreater attention was focused on the context, resultingin the design of a customised intervention programme.Our aim was not to develop a design model that definespriority needs and the relevant content of such an inter-vention programme, but to show how to identify theseneeds and the key aspects in order to design the inter-vention. Moreover, by basing the design of the interven-tion programme on the local context, the risk is that theintervention designed may not be reproducible else-where. This is a key-aspect for large-scale circulation

[66]. Future research should investigate this balance be-tween designing a pilot study and the ability to transferit.

ConclusionDespite limitations, our investigation reveals two find-ings about the RPMS content and the likelihood of en-countering various issues relating to the implementationprocess during the design phase. Firstly, the RPMSprogramme is not only a technological innovation,something which is often outlined, but also an organisa-tional innovation. This means that it is important to ac-knowledge the use of IT in conjunction with humanpractices and in a specific context (i.e. in our case, the 2NNs and their relationship with other healthcare profes-sionals). Secondly, this study confirms that the designphase of RPMS and, more generally, of any organisa-tional intervention, must not be overlooked. As regardsthe methodological aspects of designing complex health-care interventions, we wish to emphasise the fact that in-corporation of the local context and relevant processevaluation are crucial in order to design an appropriateintervention programme and promote acceptance byusers. Research programmes must therefore include therelevant dedicated stages. These preliminary phases war-rant a constant review of the intervention content inorder to ensure that it is fit for purpose in the given con-text. This should help to increase the likelihood ofimplementing an intervention programme in the mostappropriate manner, which is a current issue in modernhealthcare delivery systems.

AbbreviationsCNIL: Commission Nationale de l’Informatique et des Libertés (CNIL) (FrenchNational Data Protection Commission); ARS Ile de France: Agence Régionalede Santé (French Regional Health Agency) –Ile-de France;CAPRI: Cancérologie Parcours Région Ile de France (Regional OncologyPathway); MRC: UK Medical Research Council; NN: Nurse Navigators;RPMS: Remote Patient Monitoring System

AcknowledgementsWe wish to thank M. Azoulay, M. Bonnamy, A. Duflot-Boukobza, M. Lacaze, D.Mathivon, Vanessa Puglisi and G.Vial, for their helpful advice on and involve-ment in the CAPRI project.

Authors’ contributionsMF and EM made substantial contributions to the study design and draftingof the manuscript. AF contributed to the study design and participated indrafting the paper. CS made a substantial contribution to preparation of themanuscript. BL performed the statistical analyses of the studies. MDPparticipated in programme development. MDP designed the study. OMdesigned the study and reviewed the final version of the manuscript. Allauthors read and approved the final manuscript.

FundingThis research is part of a programme supported by the French NationalResearch Agency, Regional Health Agency – Ile de France, PhilanthropiaLombard Odier Foundation, Novartis and Astrazeneca. The funding bodieswere not involved in the study design.

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 13 of 16

Availability of data and materialsThe data sets analysed in this study are available from the correspondingauthor on request.

Ethics approval and consent to participateStudies mentioned in this article were conducted with the approval of theClinical Trial Department at Gustave Roussy (Scientific Commission forClinical Trials) and by the competent French National Authority (CPP Ile deFrance 4– Ethics Committee) and CNIL (French National Data ProtectionCommission).All patients enrolled in the study completed a written consent form toparticipate in the study.

Consent for publicationNot applicable.

Competing interestsThe authors declare that they have no competing interests.

Author details1Capri program, Research Division, Gustave Roussy, Villejuif, France. 2I3, CRG,Ecole Polytechnique, CNRS, Palaiseau, France. 3Lorraine University, CNRS, Inria,Nancy, France. 4EHESP, Department of Health Care Management, Rennes,France. 5American Hospital, Neuilly-sur-Seine, France. 6OutpatientDepartment, Gustave Roussy, Villejuif, France.

Received: 29 January 2020 Accepted: 5 May 2020

References1. Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, et al. Overall

survival results of a trial assessing patient-reported outcomes for symptommonitoring during routine cancer treatment. JAMA. 2017;318(2):197. Cited2019 Sep 3. Available from. https://doi.org/10.1001/jama.2017.7156.

2. Denis F, Basch E, Septans A-L, Bennouna J, Urban T, Dueck AC, et al. Two-year survival comparing web-based symptom monitoring vs routinesurveillance following treatment for lung cancer. JAMA. 2019;321(3):306.Cited 2019 Oct 9. Available from. https://doi.org/10.1001/jama.2018.18085.

3. Klersy C, De Silvestri A, Gabutti G, Raisaro A, Curti M, Regoli F, et al.Economic impact of remote patient monitoring: an integrated economicmodel derived from a meta-analysis of randomized controlled trials in heartfailure. Eur J Heart Failure. 2011;13(4):450–9. Cited 2019 Oct 24. Availablefrom. https://doi.org/10.1093/eurjhf/hfq232.

4. Hummel JP, Leipold RJ, Amorosi SL, Bao H, Deger KA, Jones PW, et al.Outcomes and costs of remote patient monitoring among patients withimplanted cardiac defibrillators: an economic model based on the PREDICTRM database. J Cardiovasc Electrophysiol. 2019;30(7):jce.13934 Cited 2019Oct 24. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30938894.

5. Warrington L, Absolom K, Conner M, Kellar I, Clayton B, Ayres M, et al.Electronic Systems for Patients to Report and Manage Side Effects of CancerTreatment: Systematic Review. J Med Int Res. 2019;21(1):e10875 Cited 2020Apr 21. Available from: http://www.jmir.org/2019/1/e10875/.

6. Bodenheimer T, Wagner EH, Grumbach K. Improving primary care forpatients with chronic illness. JAMA. 2002;288(15):1909. Cited 2019 Oct 24.Available from. https://doi.org/10.1001/jama.288.15.1909.

7. Bodenheimer T, Wagner EH, Grumbach K. Improving primary care forpatients with chronic illness. JAMA. 2002;288(14):1775. Cited 2019 Oct 24.Available from. https://doi.org/10.1001/jama.288.14.1775.

8. Freeman HP. Patient navigation: a community centered approach toreducing cancer mortality. J Cancer Educ. 2006;21(1, suppl):S11–4 Availablefrom: http://www.ncbi.nlm.nih.gov/pubmed/17020496.

9. Eschiti V, Burhansstipanov L, Watanabe-Galloway S. Native Cancernavigation: the state of the science. Clin J Oncol Nurs. 2012;16(1):73–89.

10. Madhavan S, Sanders AE, Chou W-YS, Shuster A, Boone KW, Dente MA, et al.Pediatric palliative care and eHealth opportunities for patient-centered care.Am J Prev Med. 2011;40(5 Suppl 2):S208–16 Available from: http://www.ncbi.nlm.nih.gov/pubmed/21521596.

11. Canadian Diabetes Association Clinical Practice Guidelines ExpertCommittee. Canadian Diabetes Association 2008 clinical practice guidelinesfor the prevention and management of diabetes in Canada. Can J Diabetes.

2008:1–201 Available from: http://www.diabetesclinic.ca/en/pdf/CDA_cpg-2008.pdf.

12. Swedberg K, Wolf RAN, Ekman RIN. Telemonitoring in Patients withHeart Failure. N Engl J Med. 2011;36411(17) Available from: http://www.utbildning.gu.se/digitalAssets/1331/1331376_new-england-journal-letter-gpcc.pdf.

13. Hesse BW, Hanna C, Massett HA, Hesse NK. Outside the box: willinformation technology be a viable intervention to improve the quality ofCancer care? JNCI Monographs. 2010;2010(40):81–9 Available from: http://www.ncbi.nlm.nih.gov/pubmed/20386056.

14. Scalvini S, Bernocchi P, Zanelli E, Comini L, Vitacca M. Maugeri centre fortelehealth and telecare: a real-life integrated experience in chronic patients.J Telemed Telecare. 2018;24(7):500–7. Cited 2020 Apr 21. Available from.https://doi.org/10.1177/1357633X17710827.

15. Abdallah LM. Evercare nurse practitioner practice activities: similarities anddifferences across five sites. J Am Acad Nurse Pract. 2005;17(9):355–62 Cited2020 Apr 24. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16115114.

16. Tregenza L. The role of the community matron: a structured approach tocase management. Nurs Older People. 2019;31(6):28–32 Cited 2020 Apr 24.Available from: http://www.ncbi.nlm.nih.gov/pubmed/31778064.

17. Slev VN, Mistiaen P, Pasman HRW, Leeuw IMV, van Uden-Kraan CF, FranckeAL. Effects of eHealth for patients and informal caregivers confronted withcancer: A meta-review. Int J Med Inform. 2016;87:54–67 Cited 2020 Jan 21.Available from: http://www.ncbi.nlm.nih.gov/pubmed/26806712.

18. Vegesna A, Tran M, Angelaccio M, Arcona S. Remote patient monitoring vianon-invasive digital technologies: a systematic review. Telemed E Health.2017;23(1):3–17 Cited 2019 Sep 13. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27116181.

19. Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al.Designing and evaluating complex interventions to improve health care.BMJ. 2007;334(7591) Available from: http://www.bmj.com/content/334/7591/455/rapid-responses.

20. Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongsiderandomised controlled trials of complex healthcare interventions:methodological study. BMJ. 2009;339 Available from: http://www.bmj.com/content/339/bmj.b3496.

21. Hoffmann TC, Erueti C, Glasziou PP. Poor description of non-pharmacological interventions: analysis of consecutive sample ofrandomised trials. BMJ. 2013;347:f3755 Cited 2019 Oct 9. Available from:http://www.ncbi.nlm.nih.gov/pubmed/24021722.

22. Oakley A, Strange V, Bonell C, Allen E, Stephenson J. Process evaluation inrandomised controlled trials of complex interventions. BMJ. 2006;332(7538):413–6.

23. Datta J, Petticrew M. Challenges to evaluating complex interventions: acontent analysis of published papers. BMC Public Health. 2013;13(1):568.Available from:. https://doi.org/10.1186/1471-2458-13-568gm.

24. Levati S, Campbell P, Frost R, Dougall N, Wells M, Donaldson C, et al.Optimisation of complex health interventions prior to a randomisedcontrolled trial: a scoping review of strategies used. Pilot Feasibility Stud.2016;2(1):17 Available from: http://www.pilotfeasibilitystudies.com/content/2/1/17.

25. Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M.Developing and evaluating complex interventions: the new MedicalResearch Council guidance. BMJ. 2008;337(October):979–83.

26. May CR, Cummings A, Girling M, Bracher M, Mair FS, May CM, et al. UsingNormalization Process Theory in feasibility studies and process evaluationsof complex healthcare interventions: a systematic review. ImplementationSci. 2018;13(1):80. Cited 2020 Jan 21. Available from. https://doi.org/10.1186/s13012-018-0758-1.

27. Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al.Normalisation process theory: a framework for developing, evaluating andimplementing complex interventions. BMC Med. 2010;8(1):63. Availablefrom:. https://doi.org/10.1186/1741-7015-8-63.

28. Michie S, Fixsen D, Grimshaw JM, Eccles MP. Specifying and reportingcomplex behaviour change interventions: the need for a scientific method.Implementation Sci. 2009;4(1):40. Cited 2019 Sep 13. Available from. https://doi.org/10.1186/1748-5908-4-40.

29. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Processevaluation of complex interventions: Medical Research Council guidance.BMJ. 2015;350:h1258.

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 14 of 16

30. Bobrow K, Farmer A, Cishe N, Nwagi N, Namane M, Brennan TP, et al. Usingthe Medical Research Council framework for development and evaluationof complex interventions in a low resource setting to develop a theory-based treatment support intervention delivered via SMS text message toimprove blood pressure control. BMC Health Serv Res. 2018;18(1):33. Cited2019 Sep 13. Available from. https://doi.org/10.1186/s12913-017-2808-9.

31. Mir O, Ferrua M, Fourcade A, Delphine M, Duflot-Boukobza A, Dumont S,et al. Intervention combining nurse navigators (NNs) and a mobileapplication versus standard of care (SOC) in cancer patients (pts) treatedwith oral anticancer agents (OAA): results of CAPRI, a single-center,randomized phase III trial. In: American Society of Clinical Oncology, ASCOAnnual meeting; 2020.

32. Fitzmaurice C, Dicker D, Pain A, Hamavid H, Moradi-Lakeh M, MacIntyre MF,et al. The global burden of Cancer 2013. JAMA Oncol. 2015;1(4):505.

33. Mariotto AB, Yabroff KR, Shao Y, Feuer EJ, Brown ML. Projections of the costof cancer care in the United States: 2010-2020. J Natl Cancer Institute. 2011;103(2):117–28 Available from: http://www.ncbi.nlm.nih.gov/pubmed/21228314.

34. Greer JA, Amoyal N, Nisotel L, Fishbein JN, MacDonald J, Stagl J, et al. Asystematic review of adherence to oral antineoplastic therapies. Oncologist.2016;21(3):354–76 Cited 2019 Mar 8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26921292.

35. Wood L. A review on adherence management in patients on oral cancertherapies. Eur J Oncol Nurs. 2012;16(4):432–8 Available from: http://www.ncbi.nlm.nih.gov/pubmed/22051845.

36. May P, Figgins B. Oral anticancer therapy: a comprehensive assessment ofpatient perceptions and challenges. J Commun Support Oncol. 2016;14(3):112–6 Available from: http://www.oncologypractice.com/jcso/the-publication/past-issue-single-view/oral-anticancer-therapy-a-comprehensive-assessment-of-patient-perceptions-and-challenges/432e62e821db479c37e21ecd0399cdba.html.

37. Lapointe J, Minvielle É, Sicotte C. Usages des technologies de l’informationet de communication pour la coordination des soins en cancérologie : Étatdes connaissances. Quelles recommandations pour une implantationefficace ? J De Gestion Et D’économie Médicales. 2014;31:273–302 ESKA.Available from: http://www.cairn.info/article.php?ID_ARTICLE=JGEM_135_0273.

38. Girault A, Ferrua M, Lalloué B, Sicotte C, Fourcade A, Yatim F, et al. Internet-based technologies to improve cancer care coordination: current use andattitudes among cancer patients. Eur J Cancer. 2015;51(4):551–7.

39. Yatim F, Cristofalo P, Ferrua M, Girault A, Lacaze M, Di Palma M, et al.Analysis of nurse navigators’ activities for hospital discharge coordination: amixed method study for the case of cancer patients. Support Care Cancer.2017;25(3):863–8 Available from: http://www.ncbi.nlm.nih.gov/pubmed/27830394.

40. Yatim F, Cristofalo P, Minvielle E. What Are the Unmet Information Needs ofCancer Patients? A Qualitative Study. Global Journal of Health Science[Internet]. 2017;9(12):114. http://www.ccsenet.org/journal/index.php/gjhs/article/view/71550.[cited 2018 Oct 5].

41. Duflos C, Antoun S, Loirat P, DiPalma M, Minvielle E. Identification ofappropriate and potentially avoidable emergency department referrals in atertiary cancer care center. Support Care Cancer. 2017;25(8):2377–85 Cited2019 Mar 22. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28275897.

42. Tomasone JR, Brouwers MC, Vukmirovic M, Grunfeld E, O’Brien MA, UrquhartR, et al. Interventions to improve care coordination between primaryhealthcare and oncology care providers: a systematic review. ESMO Open.2016;1(5):e000077 Available from: http://www.ncbi.nlm.nih.gov/pubmed/27843639.

43. Aubin M, Giguère A, Martin M, Verreault R, Fitch MI, Kazanjian A, et al.Interventions to improve continuity of care in the follow-up of patients withcancer. In: Aubin M, editor. Cochrane database of Systematic reviews.Chichester: Wiley; 2012. p. CD007672. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22786508.

44. Ouwens M, Hulscher M, Hermens R, Faber M, Marres H, Wollersheim H, et al.Implementation of integrated care for patients with cancer: a systematicreview of interventions and effects. Int J Qual HealthCare. 2009;21(2):137–44.Available from:. https://doi.org/10.1093/intqhc/mzn061.

45. National Institutes of Health, National Cancer Institute. CommonTerminology Criteria for Adverse Events (CTCAE). Cited 2018 Sep 10.Available from: https://ctep.cancer.gov/protocolDevelopment/electronic_applications/ctc.htm.

46. Brady MC, Stott DJ, Norrie J, Chalmers C, St George B, Sweeney PM, et al.Developing and evaluating the implementation of a complex intervention:

using mixed methods to inform the design of a randomised controlled trialof an oral healthcare intervention after stroke. Trials. 2011;12(1):168. Cited2019 Mar 27. Available from. https://doi.org/10.1186/1745-6215-12-168.

47. Gervès-Pinquié C, Daumas-Yatim F, Lalloué B, Girault A, Ferrua M, FourcadeA, et al. Impacts of a navigation program based on health informationtechnology for patients receiving oral anticancer therapy: the CAPRIrandomized controlled trial. BMC Health Serv Res. 2017;17(1):133 Availablefrom: http://www.ncbi.nlm.nih.gov/pubmed/28193214.

48. Kah P, Loh B, Mchugh C, Mohile SG, Mustian K, Flannery M, et al. Usinginformation technology in the assessment and monitoring of geriatriconcology patients. Cited 2020 Apr 21; Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5878089/pdf/nihms951716.pdf.

49. Woiceshyn J, Blades K, Pendharkar SR. Integrated versus fragmentedimplementation of complex innovations in acute health care. Health CareManage Rev. 2017;42(1):76–86 Cited 2020 Jan 21. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26469705.

50. Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A,Hofmann B, et al. Making sense of complexity in context andimplementation: the Context and Implementation of Complex Interventions(CICI) framework. Implementation Sci. 2017;12(1):21 Cited 2020 Jan 21.Available from: http://www.ncbi.nlm.nih.gov/pubmed/28202031.

51. Pierce JL, Kostova T, Dirks KT. Toward a theory of psychological ownershipin organizations. Acad Manag Rev. 2001;26(2):298–310. Cited 2020 Jan 21.Available from. https://doi.org/10.5465/amr.2001.4378028.

52. Barki H, Pare G, Sicotte C. Linking it implementation and acceptance via theconstruct of psychological ownership of information technology. J InformTechnol. 2008;23(4):269–80. Cited 2020 Jan 21. Available from. https://doi.org/10.1057/jit.2008.12.

53. Marthick M, Dhillon HM, Alison JA, Cheema BS, Shaw T. An interactive webportal for tracking oncology patient physical activity and symptoms:prospective cohort study. JMIR Cancer. 2018;4(2):e11978 Cited 2020 Apr 24.Available from: http://www.ncbi.nlm.nih.gov/pubmed/30578217.

54. Passardi A, Rizzo M, Maines F, Tondini C, Zambelli A, Vespignani R, et al.Optimisation and validation of a remote monitoring system (Onco-TreC) forhome-based management of oral anticancer therapies: an Italianmulticentre feasibility study. BMJ Open. 2017;7(5):e014617 Cited 2019 Dec 9.Available from: http://www.ncbi.nlm.nih.gov/pubmed/28554917.

55. Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic reviewof the Exploration, Preparation, Implementation, Sustainment (EPIS)framework. Implementation Sci. 2019;14(1):1. Cited 2020 Jan 21. Availablefrom. https://doi.org/10.1186/s13012-018-0842-6.

56. Kini V, Ho PM. Interventions to improve medication adherence. JAMA. 2018;320(23):2461 Cited 2020 Jan 21. Available from: http://www.ncbi.nlm.nih.gov/pubmed/30561486.

57. Suchman L, Blomberg J, Orr JE, Trigg R. Reconstructing technologies associal practice. Am Behav Scie. 1999;43(3):392–408. Cited 2020 Apr 24.Available from. https://doi.org/10.1177/00027649921955335.

58. Orlikowski WJ. Using technology and constituting structures: a practice lensfor studying technology in organizations. Organ Sci. 2000;11(4):404–28.Cited 2020 Apr 24. Available from. https://doi.org/10.1287/orsc.11.4.404.14600.

59. Gherardi S. Telemedicine: a practice-based approach to technology. HumRelat. 2010;63(4):501–24. Cited 2020 Apr 24. Available from. https://doi.org/10.1177/0018726709339096.

60. Marthick M, Janssen A, Cheema BS, Alison J, Shaw T, Dhillon H. Feasibility ofan interactive patient portal for monitoring physical activity, remotesymptom reporting, and patient education in oncology: qualitative study.JMIR Cancer. 2019;5(2):e15539 Cited 2020 Apr 24. Available from: http://cancer.jmir.org/2019/2/e15539/.

61. Minvielle E. (The patient and the system) Le patient et le système : en quêted’une organisation sur-mesure : approches innovantes du parcours desanté. Paris: Seli Arsal; 2018. p. 276.

62. Ferrua M, Yatim F, Fourcade A, Lacaze MG, Mir O, Minvielle E, et al. NurseNavigators telephone and web application follow-up intervention forpatients treated with oral anticancer medication: an optimization ofoncologists' medical time. J Clin Oncol. 2019;37(15_suppl):6544. Availablefrom. https://doi.org/10.1200/JCO.2019.37.15_suppl.6544 . [Cited 2019 Dec9].

63. Edmondson AC. Teaming: how organizations learn, innovate, and competein the knowledge economy [Internet]. Wiley. Jossey-Bass; 2012. p. 352.Available from: https://www.wiley.com/en-us/Teaming%3A+How+

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 15 of 16

Organizations+Learn%2C+Innovate%2C+and+Compete+in+the+Knowledge+Economy-p-9781118216767. [cited 2020 May 7].

64. Ferrua M, di Palma M, Fourcade A, Guillet M, Mathivon D, Puglisi V, et al.Patient experience and use of an intervention combining nurse-ledtelephone and technologies for the monitoring of oral cancer medication.Ann Oncol. 2019;30(Supplement_5) Cited 2019 Dec 9. Available from.https://doi.org/10.1093/annonc/mdz272.011/5578032.

65. Minvielle E, Ferrua M, di Palma M, Fourcade A, Guillet M, Mathivon D, et al.Providing a nurse-led telephone intervention for patients treated with oralanticancer medication: Symptom management and adherence monitoring.Ann Oncol. 2019;30:v837 Cited 2020 Apr 24. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0923753419602395.

66. Schloemer T, Schröder-Bäck P. Criteria for evaluating transferability of healthinterventions: a systematic review and thematic synthesis. ImplementationSci. 2018;13(1):88. Cited 2020 Jan 21. Available from. https://doi.org/10.1186/s13012-018-0751-8.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.

Ferrua et al. BMC Health Services Research (2020) 20:434 Page 16 of 16