Special theme: .,.8&1 *&18- - ERCIM News · 35 SmartWork: Supporting Active and Healthy Ageing at Work for Office Workers by Otilia Kocsis, Nikos Fakotakis and Konstantinos Moustakas

ERCIM NEWSNumber 118 July 2019ercim-news.ercim.eu

Special theme:

DigitalHealth

Joint ERCIM Actions

ERCIM NEWS 118 July 20192 ERCIM NEWS 118 July 2019

Contents

SPECIAL tHEME

The special theme Digital Health has been coordinated by Sara Colantonio (ISTI-CNR) and Nicholas Ayache (Inria)

Introduction to the special theme4 The Digital Health Revolution

by Sara Colantonio (ISTI-CNR) and Nicholas Ayache(Inria)

6 Machine Learning Applied to Ultrasound Imaging –

The Next Step in Democratising Medical Imaging

by Anne-Laure Rousseau (Assistance Publique -Hôpitaux de Paris)

7 Radiomics: How to Make Medical Images Speak?

by Fanny Orlhac, Charles Bouveyron and NicholasAyache (Université Côte d’Azur, Inria)

9 Interpretable and Reliable Artificial Intelligence

Systems for Brain Diseases by Olivier Colliot (CNRS)

10 Improving Cardiac Arrhythmia Therapy with

Medical Imaging

by Maxime Sermesant (Inria and Université Côted’Azur)

12 VoxLogicA: a Spatial-Logic Based Tool for

Declarative Image Analysis

by Gina Belmonte (AOUS), Vincenzo Ciancia (ISTI-CNR), Diego Latella (ISTI-CNR) and Mieke Massink(ISTI-CNR)

13 New Directions for Recognizing Visual Patterns in

Medical Imaging

by Alessia Amelio (University of Calabria, Italy), LucioAmelio (independant researcher) and Radmila Janković(Mathematical Institute of the S.A.S.A., Serbia)

15 Radiomics to Support Precision Medicine in

Oncology

by Sara Colantonio (ISTI-CNR), Andrea Barucci(IFAC-CNR) and Danila Germanese (ISTI-CNR)

16 Deep-Learning Based Analyses of Mammograms to

Improve the Estimation of Breast Cancer Risk

by Francesca Lizzi (National Institute for NuclearPhysics, Scuola Normale Superiore, National ResearchCouncil, University of Pisa), Maria Evelina Fantacci(National Institute for Nuclear Physics, University ofPisa) and P. Oliva (National Institute for NuclearPhysics, University of Sassari)

18 Content-Based Analysis of Medical Image Data for

Augmented Reality Based Health Applications

by Andrea Manno-Kovacs (MTA SZTAKI / PPKEITK), Csaba Benedek (MTA SZTAKI) and LeventeKovács (MTA SZTAKI)

19 Artificial Intelligence: Understanding Diseases that

People Cannot Understand?

by Marleen Balvert and Alexander Schönhuth (CWI)

Editorial Information

ERCIM News is the magazine of ERCIM. Published quarterly, it reports

on joint actions of the ERCIM partners, and aims to reflect the contribu-

tion made by ERCIM to the European Community in Information

Technology and Applied Mathematics. Through short articles and news

items, it provides a forum for the exchange of information between the

institutes and also with the wider scientific community. This issue has a

circulation of about 6,000 printed copies and is also available online.

ERCIM News is published by ERCIM EEIG

BP 93, F-06902 Sophia Antipolis Cedex, France

+33 4 9238 5010, [email protected]

Director: Philipp Hoschka, ISSN 0926-4981

Contributions

Contributions should be submitted to the local editor of your country

Copyright�notice

All authors, as identified in each article, retain copyright of their work.

ERCIM News is licensed under a Creative Commons Attribution 4.0

International License (CC-BY).

Advertising

For current advertising rates and conditions, see

https://ercim-news.ercim.eu/ or contact [email protected]

ERCIM�News�online�edition:��ercim-news.ercim.eu/

Next�issue:�

October 2019, Special theme: Smart Things Everywhere

Subscription

Subscribe to ERCIM News by sending an email to

[email protected] or by filling out the form at the ERCIM

News website: https://ercim-news.ercim.eu/

Editorial�Board:

Central editor:

Peter Kunz, ERCIM office ([email protected])

Local Editors:

Austria: Erwin Schoitsch ([email protected])

Cyprus: Georgia Kapitsaki ([email protected]

France: Christine Azevedo Coste ([email protected])

Germany: Alexander Nouak ([email protected])

Greece: Lida Harami ([email protected]),

Athanasios Kalogeras ([email protected])

Hungary: Andras Benczur ([email protected])

Italy: Maurice ter Beek ([email protected])

Luxembourg: Thomas Tamisier ([email protected])

Norway: Monica Divitini, ([email protected])

Poland: Hung Son Nguyen ([email protected])

Portugal: José Borbinha ([email protected])

Sweden: Maria Rudenschöld ([email protected])

Switzerland: Harry Rudin ([email protected])

The Netherlands: Annette Kik ([email protected])

W3C: Marie-Claire Forgue ([email protected])

Cover�illustrtation: source: Shutterstock.

ERCIM NEWS 118 July 2019

37 WellCo: Wellbeing and Health Virtual Coach

by Vlad Manea (University of Copenhagen) andKatarzyna Wac (University of Copenhagen andUniversity of Geneva)

39 A Personalisation Platform for Older Adults with

Mild Cognitive Impairments

by Marco Manca, Parvaneh Parvin, Fabio Paternò,Carmen Santoro and Eleonora Zedda (ISTI-CNR)

40 Technological Memory Aids for Neurodegenerative

Diseases and the AuDi-o-Mentia Approach

by Eleni Boumpa and Athanasios Kakarountas(University of Thessaly)

42 PAPAYA: A Platform for Privacy Preserving Data

Analytics

by Eleonora Ciceri (MediaClinics Italia), MarcoMosconi (MediaClinics Italia), Melek Önen(EURECOM) and Orhan Ermis (EURECOM)

43 Resilient Network services for Critical mHealth

Applications over 5G Mobile Network Technologies

by Emmanouil G. Spanakis and Vangelis Sakkalis(FORTH-ICS)

33

20 The Genetic Diversity of Viruses on a Graphical

Map: Discovery of Resistant and Virulent Strains

by Alexander Schönhuth (CWI and Utrecht University)and Leen Stougie (CWI and VU Amsterdam)

22 Improved Antibody Optimisation for Tumour

Analysis Through the Combination of Machine

Learning with New Molecular Assay

by Anna Fomitcheva Khartchenko (ETH Zurich, IBMResearch – Zurich), Aditya Kashyap and Govind V.Kaigala (IBM Research – Zurich)

23 AI Enables Explainable Drug Sensitivity Screenings

by Matteo Manica, Ali Oskooei, and Jannis Born (IBMResearch)

25 Combining Predictive and Prescriptive Analytics to

Improve Clinical Pathways

by Christophe Ponsard and Renaud De Landtsheer(CETIC)

26 A Holistic Clinical Decision Support System for

Diagnosis of Dementia

by Mark van Gils (VTT)

28 Connecting People, Services, and Data for

Continuity of Care

by Fulvio Patara and Enrico Vicario (University ofFlorence)

29 High Quality Phenotypic Data and Machine

Learning Beat a Generic Risk Score in the

Prediction of Mortality in Acute Coronary

Syndrome

by Kari Antila (VTT), Niku Oksala (TampereUniversity Hospital) and Jussi A. Hernesniemi(Tampere University)

31 Digital Health Interoperability as a Tool Towards

Citizen Empowerment

by Dimitrios G. Katehakis and Angelina Kouroubali(FORTH-ICS)

32 Towards VNUMED for Healthcare Research

Activities in Vietnam

by Chau Vo, (Ho Chi Minh City University ofTechnology, Vietnam National University), Bao Ho(John von Neumann Institute, Vietnam NationalUniversity) and Hung Son Nguyen, University ofWarsaw

33 Empowering Distributed Analysis Across Federated

Cohort Data Repositories Adhering to FAIR

Principles

by Artur Rocha, José Pedro Ornelas, João CorreiaLopes, and Rui Camacho (INESC TEC)

35 SmartWork: Supporting Active and Healthy Ageing

at Work for Office Workers

by Otilia Kocsis, Nikos Fakotakis and KonstantinosMoustakas (University of Patras)

RESEARCH AND INNovAtIoN

This section features news about research activities andinnovative developments from European researchinstitutes

46 A Contractarian Ethical Framework for Developing

Autonomous Vehicles

by Mihály Héder (MTA SZTAKI)

48 A Language for Graphs of Interlinked Arguments

by Dimitra Zografistou, Giorgos Flouris, TheodorePatkos, and Dimitris Plexousakis (ICS-FORTH)

49 Distortion in Real-World Analytic Processes

by Peter Kieseberg (St. Pölten University of AppliedSciences), Lukas Klausner (St. Pölten University ofApplied Sciences) and Andreas Holzinger (MedicalUniversity Graz).

ANNouCEMENtS, IN bRIEf

45 ERCIM “Alain Bensoussan Fellowship Programme

50 GATEKEEPER - Smart Living Homes for People at

Health and Social Risks

51 HORIZON 2020 Project Management

51 In Memory of Cor Baayen

51 Dagstuhl Seminars and Perspectives Workshops


Special theme: Digital Health

the success of deep learning techniquesin processing ultrasound imaging data todiagnose kidney diseases, and detect uri-nary track blocks as well as kidneycancer. She has gathered a multidiscipli-nary team of engineers and health pro-fessionals, NHance, to develop applica-tions that may leverage the great poten-tial of machine learning to overcome thecurrent difficulties in ultrasound reading(p. 6).

The work by Orlhac et al. (p.7) intro-duces radiomics, an emerging disciplineconsisting of the extraction of a largenumber of quantitative parameters frommedical images, whose mining may leadto predictive models for prognosis andtherapy-response that support evidence-based clinical decision-making. In hiswork, Colliot (p. 9) discusses some ofthe most cogent issues pertaining to theinterpretability and reliability of artifi-cial intelligence algorithms whenapplied to medical image analyses.Sermesant illustrates how the rich anddetailed information contained in mag-netic resonance and computed tomog-raphy cardiac images may be used toreconstruct a 3D heart model to providethe physicians with visual data that maybe navigated during an intervention andcan even predict the result of an inter-vention (p. 10).

Belmonte et al. (p. 12) introduce a novelapproach to medical image analysisbased on the recent advances in spatiallogics. VoxLogicA is the tool developedby the authors to process both 2D and3D imaging data. The work by Amelio etal. addresses the problem of medicalimage registration to track the evolutionof a disease over time (p. 13). Theauthors introduce a new similaritymeasure to register CT brain images andmonitor the evolution of brain lesions.

Introduction to the Special Theme

the Digital Health Revolution

by Sara Colantonio (ISTI-CNR) and Nicholas Ayache (Inria)

This special theme focuses on research advances and perspectives in digital health, offering a glimpse

into the diverse range of topics and challenges that the research community is facing in the field.

The convergence of science and tech-nology in our burgeoning digital era isdriving a complete transformation ofhealth, medicine and care paradigms,which aim to improve people’s long-term wellbeing and quality of life.“Digital health” arises where value-based and system medicine meets digitalinnovations, harnessing the potential ofnew technologies to make the patient thepoint of care and modernise the deliveryof health and care services.

Society’s ageing population, thegrowing prevalence of chronic diseasesand multimorbidities, and the shortageof clinicians and care personnel are bigchallenges for health and care systems.Society expects improved quality andexperience of health services, integratedcare systems, and greater equality ofaccess to health and care [L1].

Digital health can offer solutions byleveraging recent advances in com-puting methodologies and engineeringas well as capitalising on the fertile,multidisciplinary environment and theincreasing availability of data. The com-bination of artificial intelligence andmachine learning, big data analytics andcomputer vision techniques with multi-omics research, portable diagnostics,wearables and implantable sensors ishelping us understand biological, socialand environmental processes thatunderlie disease onset. Technologiessuch as mHealth (mobile health), aug-mented reality, robotics and 3D printing,may enable more precise diagnoses,interventions and personalised follow-up programmes, and help improve effi-ciency, resilience and sustainability ofhealthcare systems.

Digital health emerges as key to ensurethe shift towards the 4P paradigm, which

aims at more predictive, preventative,personalised and participatoryapproaches to health and care [1].However, its realisation requires vastamounts of curated and high-qualitydata, regulations for privacy, data own-ership, security and liability issues, stan-dardisation and interoperability. Trustedsolutions should ensure reliability inhandling patient safety.

This special theme features research,providing significant examples of thegreat potential of effective digital healthsystems, and a panorama of theimpactful and vibrant community that isactively working in the field. The pre-sented approaches target: • clinicians and health professionals, to

empower them through the provisionof actionable insights for faster andmore accurate diagnoses and prog-noses, as well as for more precise,patient-tailored treatments, follow-ups and assistance.

• health and care platforms to ensureaccessible, interoperable and sustain-able systems.

• individuals, patients and informalcaregivers, to make them active play-ers in the management of health, viatimely and targeted prevention andassistance strategies.

One of the well-established fields fordigital health solutions is medical imageanalysis and understanding, whoserecent advances is thrilling the researchcommunity. The recent successes of arti-ficial intelligence in computer vision ispromisingly heading towards systemsthat may reduce the workload of radiolo-gists in intensive error-prone manualtasks, and exploit the rich content ofimaging data into disease phenotypes formore accurate diagnoses. In her contri-bution, Anne-Laure Rousseau illustrates

ERCIM NEWS 118 July 2019 5

Barucci et al. illustrate the potential ofradiomic analyses to provide prognosticproxy data in the oncological domain (p.15). The work by Lizzi et al. overviewsdeep-learning based analyses of mam-mographic images to assess the risk ofbreast cancer (p.16). Manno-Kovacs etal. explore the possibility of using aug-mented reality for the visualisation of3D imaging data (p. 18).

Advances in artificial intelligence andmachine learning are of great help toprocess multiomics data and, thus,gather an understanding of the processesthat underlie disease onset and itsresponse to therapy. In this context, thework by Balvert and Schönhuth (p. 19)presents a deep learning approach tounravel the complex architecture ofserious diseases such as the Alzheimer’sand amyotrophic lateral sclerosis.Schönhuth and Stougie address theproblem of decoding the genetic diver-sity of viruses, in order to detect poten-tial resistant and virulent strains throughthe construction of virus variationgraphs (p. 20). Manica et al. exploredeep learning architectures to predictdrug sensitivity (p. 23). They adopt fea-ture saliency techniques to explain theresults provided by the network, thusidentifying the genes involved.Instrumental to the histopathologyanalyses is the identification of theoptimal stain of the specimen for thespecific tissue under examination.Khartchenko et al. (p. 22) use machinelearning to predict the quantity of stainin accordance to the quality of theresulting assay.

Supporting clinical decision making is acentral idea of digital health, since it per-mits clinicians to provide evidence-based care for the patients and reducedtreatment costs. Ponsard and DeLandtsheer (p. 25) combine predictiveand prescriptive analytics to optimisethe organisation of care processes andthe definition of clinical pathways. Thework by van Gils combines data derivedfrom various sources (imaging, lab data,neuropshychological tests) into a singletool that supports the differential diag-nosis of dementia (p. 26).

Efficient and integrated ICT infrastruc-tures have a crucial role to support clin-ical practice by delivering the right dataand information at the right time to theright end-user (including citizens), thusbreaking data isolation and fosteringprecision medicine and care. Much workhas been done in the field, but someimportant issues remain open (e.g., datainteroperability and curation) and newones are emerging (e.g., integrating andmanaging citizens’ generated data andgranting them data access rights).Katehakis and Kouraubali (p. 31) dis-cuss the advances of the EuropeanInteroperability Framework developedin response to the needs and expecta-tions that emerged from the last openconsultation of the EuropeanCommission. Patara and Vicario presentan adaptable Electronic Health Record(EHR) system (p. 28). Antila et al. (p.29) demonstrate that the availability ofdata collected into EHRs may be suc-cessfully used to develop individuals’phenotypic models that may performbetter than the risk scores currently inuse in routine practice in mortality pre-diction. The work by Chou Vo et al.reports on recent advances in Vietnam inthe deployment of hospital EHRs andtheir integration into a unified database,named VNUMED (p. 32). Rocha et al.debate the use of FAIR principles totackle data integration and harmonisa-tion while preserving privacy (p. 33).

For preventative strategies to have along term impact, it is essential that indi-viduals are empowered as active partici-pants in their health management. Thisis an area that continues to be a focus inthe digital transformation of health andcare. Remote and self-monitoring sys-tems are now being designed for workenvironments to sustain workability ofthe ageing workforce, as Kocsis and col-leagues present in their contribution (p.35). Their solution relies on a flexibleand integrated worker-centric AI Systemto support health, emotional and cogni-tive monitoring as well as working taskadaptation. User engagement remains acrucial feature to ensure the long-termimpact of these personal applications.Manea and Wac (p. 37) describe theirapproach to engaging behavioural

change with a health and well-being per-sonal virtual coach, developed in theWellCo project. Sustained quality of lifeand independence are the main goal ofthe contributions by Paternò and col-leagues (p. 39) and Boumpa andKakarountas (p. 40), who propose adapt-able assistive technologies to supportpeople affected by cognitive impair-ments.

The final two papers of the specialtheme propose approaches to importantissues in digital health. Ciceri et al.tackle privacy preservation when devel-oping and deploying data analyticstools, capitalising on a platform devel-oped in the PAPAYA project (p. 42).Spanakis and Sakkalis propose resilientnetwork services for ensuring the con-tinuous availability of Internet connec-tion to mHealth applications, thus safe-guarding their reliability and accept-ability (p. 43).

In conclusion, this special issue providesthe readers with a vibrant illustration ofa sample of the multi-disciplinaryresearch activities which underpin theupcoming revolution of digital health.

Link:

[L1] https://kwz.me/hyV

Reference:

[1] M. Flores, G. Glusman, K.Brogaard, N.D. Price, L. Hood: “P4medicine: how systems medicinewill transform the healthcare sectorand society”, Per Med., 10 (6)(2013), pp. 565-576.

Please contact:

Sara Colantonio, ISTI-CNR, [email protected]

Nicholas Ayache, Inria, [email protected]

The new techniques of computer visioncan be seen as a modern form of automa-tion [1]: after automating the legs (thephysical work made possible with themotors), the hands (precision work madepossible by robots in the industry), weautomate the eyes (the detection of pat-terns in images). The computer visionalgorithms are skilled at specific tasks:imagine having with you not one or two,but a thousand medical students who canquickly perform tasks that are not neces-sarily complex but very time consuming.This will change a lot of things: it willreduce the time necessary to complete animaging exam but it will also change thevery nature of these exams by changingthe cost efficiency of certain procedures.It is the combination of the doctor andthe machine that will create a revolution,not the machine alone.

I gathered together a group of engi-neers and healthcare workers, the

NHance team [L1], to apply deeplearning algorithms to ultrasoundimaging. Ultrasound imaging has seena most impressive growth in recentyears and over 20 medical and sur-gical special t ies have expressedinterest in its use as a hand-carriedultrasound tool. According to WHO,two thirds of the world’s populationdoes not have access to medicalimaging, and ultrasound associatedwith X-ray could cover 90 % of theseneeds. The decrease in hardwareultrasound prices allows its diffusion,and it is no longer access to equip-ment but the lack of training thatlimits its use [2].

Indeed, learning ultrasound is cur-rently complex and not scalable.Machine learning could be particu-larly useful to alleviate the constraintsof ultrasound, and further democratisemedical imaging around the world.

Using algorithms to detect kidney dis-eases is one example of this. We trainedour algorithms on 4,428 kidney images.To measure our model’s performancewe use the receiver operating character-istic curves (ROC). The areas under theROC curves (AU-ROC) quantify thediscrimination capabilities of the algo-rithms for the detection of a diseasecompared to the diagnosis made byNhance medical team. The higher theAU-ROC, the better the model is at dis-tinguishing between patients with dis-ease and no disease. The modele whosepredictions are 100% wrong has an AU-ROC of 0.0. On the contrary, an excel-lent classifier has an AU-ROC ofaround 1. In our study; the AU-ROC fordifferentiating normal kidneys fromabnormal kidneys is 0.9.

We have trained algorithms to solve anemergency problem: for every acuterenal failure, the physician needs to



Machine Learning Applied to ultrasound Imaging –

the Next Step in Democratising Medical Imaging

by Anne-Laure Rousseau (Assistance Publique - Hôpitaux de Paris)

Machine learning has made a remarkable entry into the world of healthcare, but there remain some

concerns about this technology. According to journalists, a revolution is upon us: One day the first artificial

intelligence robot receives its medical degree, the next, new algorithms have surpassed the skill of medical

experts. It seems that any day now, medical doctors will be unemployed, superseded by the younger siblings

of Siri. But having worked on both medical imaging and machine learning, I think the reality is different,

and, at the risk of disappointing some physicians’ colleagues who thought the holidays were close, there is

still work for us doctors for many decades. The new technology, in fact, offers a great opportunity to

enhance health services worldwide, if doctors and engineers collaborate better together.

This�concept-image�depicts�how�physicians

could�use�deep�learning�algorithms�in�their

future�practice.

know if the urinary tract is blocked, andultrasound is a good way to performthat diagnosis (a kidney state calledhydronephrosis). AU-ROC for identi-fying hydronephrosis is 0.94 in ourstudy.

We have trained our algorithms todetect kidney cancer. If kidney canceris diagnosed when the patient hassymptoms, such as blood in the urine orlow back pain, it means in most casesthat the cancer has already evolved andenlarged outside the kidney and it willbe extremely difficult to cure, so high-lighting a kidney cancer when it isasymptomatic can be a determinantfactor for the patient. AU-ROC foridentifying parenchymal lesion of thekidney is 0.94 in our study.

We have applied our research to otherorgans, such as the liver. The incidenceof and mortality from liver diseasesgrows yearly. Our deep learning algo-rithms present an AU-ROC of 0.90 for

characterisation of a benign liver lesionand up to 0.90 for a malignant liverlesion. For medical experts, theAUROC for discriminating the malig-nant masses from benign masses was0.724 (95 % CI, 0.566-0.883, P =0.048) according to Hana Park et al.

These results are very encouraging andshow that in the near future our algo-rithms could play an important role inassisting doctors with diagnoses. Ournext step is to scale these results in col-laboration with the Epione team led byNicholas Ayache and Hervé Delingetteat INRIA. We will work on all theabdominal organs in partnership withpublic assistance hospitals of Paris(APHP) that puts 1.3 million exams atour disposal for our project.

It’s time to take cooperation betweenmedical teams and engineering teams tothe next level, in order to build bettertools to enhance healthcare foreveryone.

Link:

[L1] http://www.nhance.ngo/

References:

[1] B. Evans: “10 Year Futures (Vs.What’s Happening Now) [Internet],Andreessen Horowitz, 6 Dec 2017https://kwz.me/hy9

[2] C.L. Moore, J.A. Copel: “Point-of-Care Ultrasonography”, N. Engl. J.Med. 24 Feb. 2011, 364(8):749‑757.doi:10.1056/NEJMra0909487.

[3] S. Shah, BA Bellows, AA Adedipe,JE Totten, BH Backlund, D Sajed:Perceived barriers in the use ofultrasound in developing countries,Crit. Ultrasound J. 19 June 2015; 7. doi: 10.1186/s13089-015-0028-2.

Please contact:

Anne-Laure Rousseau, President of theNGO NHance, European HospitalGeorges Pompidou and Robert DebréHospital, Paris, [email protected]


Medical images are now routinelyacquired during the care pathway and playan important role in patient management.However, medical images are still largelyunder-exploited, using mostly visualassessment and/or the measurement ofvery few quantitative features available inclinical practice. To extract more informa-tion from medical images, a new field,radiomics, has successfully developedsince 2010, with almost 1000 publicationsnow using “radiomics” in PubMed, morethan 30 % of which were published duringthe first half of 2019. Just as “genomics”,“proteomics” and “metabolomics” referto the study of large sets of biologicalmolecules, “radiomics” refers to the auto-matic extraction of large sets of quantita-tive features from medical images.Radiomic features can be derived fromknown mathematical expressions andreflect, for instance, the distribution ofgrey-levels, the shape of a volume ofinterest or the texture of the signal within

that volume. More recently, in addition ofthese handcrafted features, an infinitenumber of “deep features” can beextracted from intermediate layers ofconvolutional neural networks.

In oncology, encouraging results usingradiomic features have been publishedto predict biological characteristics oflesions, patient response to therapy oroverall survival, for several cancertypes and imaging modalities (seeFigure 1). In the coming years,radiomics enhanced by artificial intelli-gence techniques will certainly play amajor role in patient management and inthe development of 4P medicine (predic-tive, preventive, personalised and partic-ipatory).

Epione [L1] is a research team affiliatedwith Université Côte d’Azur, Inria(Sophia-Antipolis, France). The long-term goal of the team is to contribute to

the development of the e-patient (digitalpatient) for e-medicine (digital medi-cine). With the ongoing digital revolu-tion in medicine and the need to analyseand interpret more and more high-dimensional data, the team has devel-oped, for instance, a subspace discrimi-nant analysis method which performs aclass-specific variable selection throughBayesian sparsity, called sparse high-dimensional discriminant analysis(sHDDA) [1]. We demonstrated theinterest of sHDDA for the radiomicanalysis of computed tomography (CT)images to distinguish between lesionsubtypes in lung cancer [1] or to iden-tify triple-negative breast lesions basedon positron emission tomography(PET) images [2]. Below we presenttwo ongoing projects in radiomics incollaboration with clinical partnerships.

Radiomics: How to Make Medical Images Speak?

by Fanny Orlhac, Charles Bouveyron and Nicholas Ayache (Université Côte d’Azur, Inria)

Radiomics is the automatic extraction of numerous quantitative features from medical images and using

these features to build, for instance, predictive models. It is anticipated that Radiomics enhanced by AI

techniques will play a major role in patient management in a clinical setting. To illustrate developments

in this field, we briefly present two ongoing projects in oncology.



Management of glioblastomasAfter an initial standard treatment, andin the case of clinically suspected recur-rence, the differentiation between pro-gression and radiation-induced necrosisfor patients with a glioblastoma is oftendifficult based on magnetic resonance(MR) images. The use of PET imagescombined with a specific radiotracer(18F-FDOPA) improves the differentialdiagnosis but is still not completelyaccurate. In collaboration with Prof. J.Darcourt, Dr O. Humbert and Dr. F.Vandenbos from the Centre AntoineLacassagne [L2], we studied the rele-vance of radiomic features extractedfrom PET images to distinguishbetween progression and radiation-induced necrosis.

Using a retrospective cohort of 78patients with a glioblastoma, we createdparametric images resulting from thesubtraction of two static PET scans per-formed 20 and 90 minutes after injec-tion of the radiotracer (PETsub=PET90– PET20). Based on these new imagesand for each suspicious lesion, weextracted 43 radiomic features usingLIFEx software [L3] including conven-tional features used in clinical practice,as well as histogram, shape and textureindices. We demonstrated that, thanks toa machine learning approach designedfor low-sample size/high-dimensionaldata, the High-Dimensional DiscriminantAnalysis method (HDDA), it is possibleto distinguish between progression andradiation necrosis with better perform-ance than visual assessment [3]. In ourcohort, the visual interpretation usingthe Lizarraga Scale led to a YoudenIndex (Y=Sensitivity+Specificity-1) of0.27 (Sensitivity = 97 %, Specificity =

30 %). Based on the radiomic analysisof the parametric images resulting fromthe evolution of radiotracer uptake, theYouden Index was 0.45 (Sensitivity =65 %, Specificity = 80 %). Additionalstudies are ongoing to validate theseresults on an independent cohort and totest if including additional featuresextracted from MR images could fur-ther improve the performance.

Prediction of treatment responseIn oncology, a major challenge for clini-cians is to identify the right treatmentfor the right patient at the right time. Toassist them in this task, our goal is todevelop radiomic signatures to predictthe patient’s response to therapy. In col-laboration with Dr T. Cassou Mounat,Dr A. Livartowski, and Dr M. Luporsifrom Institut Curie [L4] and Dr. I. Buvatfrom Laboratoire IMIV [L5], we arefocusing on two cancer types. In lungcancer, our objective is to combineradiomic features extracted from pre-treatment PET images in order to pre-dict the response to chemotherapy. Asnearly 50 % of patients do not respondpositively, their early identificationcould allow clinicians to propose alter-native therapies, such as immuno-therapy, straight away. In breast cancer,we are developing a radiomic signatureto predict the response to neoadjuvantchemotherapy. Indeed, a completepathological response is only observedin about 20 % of patients while 10 %have stable or progressive disease afterneoadjuvant chemotherapy. The earlyidentification of non-respondingpatients or of tumours that will go ongrowing during chemotherapy wouldmake it possible to adjust therapy at bestwithout any loss of time.

Overall, the combination of radiomicanalysis and modern machine learningapproaches paves the way to betterpatient management thanks to a moreextensive exploitation of medicalimages that are already currentlyacquired during the care pathway.

Links:

[L1] https://team.inria.fr/epione/en/[L2] https://kwz.me/hyf[L3] https://www.lifexsoft.org/[L4] https://institut-curie.org/[L5] http://www.imiv.fr

References:

[1] F. Orlhac, P.-A. Mattei, C.Bouveyron, N. Ayache: “Class-specific variable selection in high-dimensional discriminant analysisthrough Bayesian Sparsity”, JChem, 2019;33:e3097.

[2] F. Orlhac, O. Humbert, T.Pourcher, L. Jing, J.-M. Guigonis,J. Darcourt, N. Ayache, C.Bouveyron: “Statistical analysis ofPET radiomic features andmetabolomic data: prediction oftriple-negative breast cancer”, JNucl Med, 2018;59:1755.

[3] F. Orlhac, A.-C. Rollet, C.Bouveyron, J. Darcourt, N. Ayache,O. Humbert: “Identification of aradiomic signature to distinguishrecurrence from radiation-inducednecrosis in treated glioblastomasusing machine learning methods ondual-point 18F-FDOPA PETimages”, J Nucl Med, 2019;60:57.

Please contact:

Fanny OrlhacUniversité Côte d’Azur, Inria, [email protected]

Figure�1:�Process�to�obtain�radiomic�signatures�in�order�to�assist�clinicians�for�patient�management.


As in different fields of medicine, AIholds great promise to assist cliniciansin the management of neurological dis-eases. However, there is still an impor-tant gap to bridge between the design ofsuch systems and their use in clinicalroutine. Two major components of thisgap are interpretability and reliability.

“Interpretability”, the user’s ability tounderstand the output provided by an AIsystem, is important for the adoption ofAI solutions by clinicians. To make AIsystems more interpretable, differentlines of work are being pursued. A firstavenue is to explain the basis of a pre-

diction based on the input features.While this is relatively straightforwardfor linear models, it is more difficult forcomplex non-linear techniques such asdeep neural networks, even thoughadvances have recently been made inthis area.

Another complementary way to make AIsystems more interpretable is to predictnot only a clinical outcome (e.g. dis-eased/healthy, lesional/non-lesional) butalso different types of medical data andmeasurements characterising a patient.For instance, in Alzheimer’s disease,one may try not only to predict the futureoccurrence of dementia, but also thefuture value of cognitive scores or future

medical images of the patient. Recently,we proposed a system to predict brainimages that are representative of dif-ferent pathological characteristics inmultiple sclerosis. In a first work [1],we designed a system that can predict aspecific type of magnetic resonanceimage (MRI), called FLAIR, from othertypes of MR images. We showed thatoverall the predicted images did a goodjob of preserving the characteristics ofthe original image. We then applied anautomatic segmentation algorithm andshowed that its results on the predictedand original images are consistent.

We further proposed an approach to pre-dict myelin content from multiple MRImodalities (Figure 1) [2]. Myelin is asubstance that wraps axons andincreases the speed of transmission ofinformation between neurons. Multiplesclerosis is characterised by the loss ofmyelin (demyelination) whose quantifi-cation is essential for tracking diseaseprogression and assessing the effect oftreatments. Myelin can be measured invivo using positron emission tomog-raphy (PET) with specific tracers.However, PET is an expensive imagingmodality and is not available in mostcentres. We showed that it is possible tosynthesise PET images from multipleMR images, which are less expensive to

acquire. The predicted image allowsaccurate quantification the amount andlocation of demyelinated areas. Theseresults will need to be confirmed onlarger, multicentric, datasets. By pro-viding both the quantified outcome(demyelination) and the predictedimage, our approach has the potential tobe more interpretable for the clinician.

Naturally, reliability is a mandatoryproperty of medical AI systems. Designof reliable systems involves many stepsfrom evaluation of prototypes to certifi-cation of products. At the stage of aca-demic research, an important compo-

nent is the ability to replicate results of agiven study. Replication is indeed a cor-nerstone of scientific progress in allareas of science. Such a processinvolves two related but distinctaspects: reproducibility, defined as theability to reproduce results based on thesame data, and replicability, the repro-duction using different data.

In a recent work, we studied theproblem of reproducibility of deeplearning approaches for assisting diag-nosis of Alzheimer’s disease from MRIdata [3]. First, we reviewed existingstudies and unveiled the existence ofquestionable practices in a substantialnumber of them. Specifically, among

Interpretable and Reliable Artificial Intelligence

Systems for brain Diseases

by Olivier Colliot (CNRS)

In artificial intelligence for medicine, more interpretable and reliable systems are needed. Here, we

report on recent advances toward these aims in the field of brain diseases.

Figure�1:�Prediction�of�myelin�content,

as�defined�from�PET�images,�using

multiple�MRI�modalities.�On�the�left,

input�MRI�modalities:�magnetisation

transfer�ratio�(MTR)�and�three�meas-

ures�computed�from�diffusion�MRI,

axial�diffusivity�(AD),�radial�diffusivity

(RD)�and�fractional�anisotropy�(FA).

On�the�right:�predicted�and�ground

truth�PET�data.�The�PET�tracer�is�the

Pittsburgh�compound�B�(PiB)�is�used

to�measure�myelin�content�in�the�white

matter�of�the�brain.



Ventricular fibrillation is a pathology inwhich cardiac electrophysiology, whichcontrols the contraction of the heart,deteriorates into chaotic behaviour. Theelectric wave that has to activate eachmuscle fibre of the heart in a coordi-nated way becomes a storm that doesnot generate an effective contraction, acondition that is deadly within minuteswithout heart massage and defibrillator.This sudden death affects 400,000people a year in Europe, which is morethan the three most deadly cancerscombined.

The occurence of ventricular fibrillationdepends largely on the extent of myocar-dial infarction: when it is not fatal, itproduces damage that can cause arrhyth-mias years later, such as fibrillation ortachycardia. Traditionally these aretreated by implanting a defibrillator,which will trigger an adequate electricshock in case of arrhythmia. But this

does not heal anything, represents amajor intervention, and the electricshocks often affect the individual’squality of life.

In recent years, another intervention hasbeen developed, catheter ablation,which allows the cardiac cells respon-sible for arrhythmias to be burnt with acatheter. However, it is a complex pro-cedure because the cardiologist only hasaccess to a very compartmental visionof the heart during these interventionsand must therefore exhaustively searchfor the right targets.

Medical imaging nowadays, however,makes it possible to obtain, in a non-invasive way, very detailed 3D informa-tion on the anatomy and the cardiacstructure of these patients. Magneticresonance imaging and computedtomography now have spatial resolu-tions of the order of a millimetre. But

these data are not currently available tothe interventional cardiologist. Theacquired images are interpreted by theradiologist and the cardiologist receivesa report, but the 3D data cannot be usedby the catheter systems.

A new technology developed betweenInria Sophia Antipolis and the IHULiryc in Bordeaux makes it possible toextract the important information fromthe 3D images of the patient's heart andto represent them in the form of meshescompatible with the interventionaltools. The cardiologist can thereforemanipulate his catheter while visual-ising the locations of the sensitive struc-tures to avoid, and the areas to aim for.This accelerates the process, improvessafety and increases the success rate [1].This technology has been tested on hun-dreds of patients around the world andhas been transferred to the start-up com-

Improving Cardiac Arrhythmia therapy with

Medical Imaging

by Maxime Sermesant (Inria and Université Côte d’Azur)

With medical imaging’s ability to provide a high level of detail about cardiac anatomy and pathology,

it is high time for such information to be used during interventions. Technology to achieve this is now

being made available to every cardiologist.

[2] W. Wei, E. Poirion, B. Bodini, S.Durrleman, N. Ayache, B. Stankoff,O. Colliot: “Learning MyelinContent in Multiple Sclerosis fromMultimodal MRI throughAdversarial Training, in: Proc.MICCAI - Medical ImageComputing and Computer AssistedIntervention, Springer, 2018.

[3] J. Wen, E. Thibeau-Sutre, J.Samper-Gonzalez, A. Routier, S.Bottani, S. Durrleman, N. Burgos,O. Colliot: “Convolutional NeuralNetworks for Classification ofAlzheimer’s Disease: Overviewand Reproducible Evaluation”,2019. ArXiv190407773 Cs EessStat.

Please contact:

Olivier ColliotCNRS, Inria, Inserm, SorbonneUniversity, Brain and Spine Institute Paris, [email protected]

the 32 studies, half of them potentiallyintroduced data leakage (the use ofsome information from the test setduring training). Data leakage was clearin six of them and possibly present in 10others which had insufficient detailsabout their validation procedure. This isa serious problem, particularly consid-ering that all these studies have under-gone peer-review. Given these defectsin the validation procedure, it isunlikely that the very high perform-ances reported in these studies would bereplicated by others. Moreover, many ofthese works were not reproduciblebecause the code and data were notmade available.

We thus proposed a framework forreproducible experiments on machinelearning for computer-aided diagnosisof AD [3]. The framework was com-posed of standardised data managementtools for public data, image prepro-cessing pipelines, machine learning

models and validation procedure. It isopen-source and available on github[L1]. We applied the framework to com-pare the performance of different con-volutional neural networks and providea baseline to which future works can becompared. We hope that this work willbe useful to other researchers and pavethe way to reproduce research in thefield.

Link:

[L1] https://github.com/aramis-lab/AD-ML

References:

[1] W. Wei, E. Poirion, B. Bodini, S.Durrleman, O. Colliot, B. Stankoff,N. Ayache: “Fluid-attenuatedinversion recovery MRI synthesisfrom multisequence MRI usingthree-dimensional fullyconvolutional networks formultiple sclerosis”, J Med Imaging6:27, 2019. DOI:10.1117/1.JMI.6.1.014005, 2019


pany inHEART, created in 2017 to com-mercialise this tool [L1].

However, to implement this technology,it was necessary to tackle various scien-tific and technical challenges:

• It is necessary to develop robust andfast image processing algorithms.Speed is required to integrate into theclinical workflow without addingadditional time. Robustness is crucialfor the results to be relevant even indifficult cases, for example if theimage is of lower quality. The valida-tion / correction step by the user mustnot take too much time compared tothe time it would take to do every-thing manually, otherwise the algo-rithm loses its relevance.

• A key step is segmentation, whichextracts important structures from theimage. This is one of the major stepsbecause the accuracy of the informa-tion provided to the cardiologistdepends on it. The method used isbased on deep learning [2] but allowsthe user to correct the result and vali-date it.

• It also requires image registrationalgorithms, which allow the matchingof information from several modali-ties, such as MRI and CT. MRI canbe used to image fibrosis and there-fore the result of infarction, but notsmall structures such as coronaries,while CT does it very well. The abil-ity to merge these modalities allows amore complete vision of the heart.Again, there are automatic tech-niques, but it is important to obtainthe robustness necessary for clinicaluse, and that the user can guide thealgorithm, including points of interestin each of the images.

• In addition, these algorithms must beintegrated into software that can beused by a non-specialist, because inorder to allow wide-scale use, theend-user must not be a specialist inmedical imaging, nor a radiologist ora cardiologist. This is an ergonomicchallenge so that the sequence ofsteps is natural [L2].

• Finally, we must develop an intuitivevisualisation of the different meshesgenerated because they will be addedto the substantial existing data thatthe cardiologist will have to interpret

during the intervention. We are not allequal in 3D visualisation, and theresults must not be confusing, so it isimportant to interact with cardiolo-gists to optimise this aspect.

On these different aspects, the contribu-tion of computing is crucial, and the sci-entific challenge is notably to succeedin making the algorithms work onimages coming from any hospital in theworld, with different acquisition proto-cols and very variable image qualities.

A related area of research is the mathe-matical modelling of the heart, whichcould non-invasively predict ablationtargets by simulating different electricalpropagations and ablation strategiesbefore the procedure. This is also part ofthe collaborative scientific programbetween IHU Liryc and Inria and isbased on the image analysis outlinedabove. The challenge here is to succeedin quickly and robustly customisingsuch mathematical models to a patient'simages in order to generate the corre-sponding predictions [3].

Computer science allows this conver-gence of domains (imaging, model-ling, catheters) and thus creates newand exciting possibil i t ies forimproving the success and safety ofcardiac procedures.

Links:

[L1] https://www.inheart.fr[L2] https://kwz.me/hyw

References:

[1] S. Yamashita, H. Cochet, F. Sacher,S. Mahida, B. Berte, D. Hooks, J.-M. Sellal, N. Al Jefairi, A.Frontera, Y. Komatsu, H. Lim, S.Amraoui, A. Denis, N. Derval, M.Sermesant, F. Laurent, M. Hocini,M. Haissaguerre, M. Montaudon, P.Jais: “Impact of New Technologiesand Approaches for Post-Myocardial Infarction VentricularTachycardia Ablation DuringLong-Term Follow-Up”,Circulation- Arrhythmia andElectrophysiology, 9(7), 2016.

[2] S. Jia, A. Despinasse, Z. Wang, H.Delingette, X. Pennec, P. Jaïs, H.Cochet, M. Sermesant:“Automatically Segmenting theLeft Atrium from Cardiac ImagesUsing Successive 3D U-Nets and aContour Loss. In Statistical Atlasesand Computational Modeling ofthe Heart (STACOM”), LNCS,Springer, 2018.

[3] N. Cedilnik, J. Duchateau, R.Dubois, F. Sacher, P. Jaïs, H.Cochet, M. Sermesant: “FastPersonalized ElectrophysiologicalModels from CT Images forVentricular Tachycardia AblationPlanning”, EP-Europace, 20, 2018.

Please contact:

Maxime SermesantInria and Université Côte d’Azur,France+33 4 92 38 78 [email protected]

Figure�1:�3D�rendering�of�the

cardiac�atria�and�ventricles

(white),�veins�(blue),

coronaries�(red),�phrenic

nerve�(green)�from�CT�and

fibrosis�quantification�(yellow

to�red)�from�MRI.�This

detailed�information�provides

structures�to�avoid�(vessels,

nerves)�and�areas�to�treat

(fibrosis)�when�performing

catheter�ablation�of

arrhythmias.

VoxLogicA is a new tool to performimage analysis in a declarative way,based on the latest developments in thetheory of spatial logics [L1]. The toolwas developed as part of a collaborationbetween the "Formal Methods andTools" group of ISTI-CNR and theDepartment of Medical Physics of theAzienda Ospedaliera UniversitariaSenese (AOUS).

VoxLogicA can be used for 2D (gen-eral-purpose) imaging, and for analysing3D medical images, which has so farbeen the most promising applicationscenario. VoxLogicA has been designedwith an emphasis on simplicity, and witha focus on producing explainable andimplementation-independent results. AVoxLogicA session consists of a textual

specification of image analysis,employing a combination of spatial fea-tures (distance between regions, orinter-reachability) with texture simi-larity, statistical, and imaging primi-tives.

VoxLogicA is publicly distributed, freeand open source software (see link). Atits heart lies a "model checker"; a veryefficient computation engine for logicalqueries, exploiting advanced tech-niques, such as memoization and multi-threading, to deliver top-notch perform-ance.

VoxLogicA sessions are written using adeclarative logical language, “ImageQuery Language” (ImgQL), inspired bythe very successful "Structured Query

Language" (SQL) for databases, butwith strong mathematical foundationsrooted in the area of spatial logics fortopological (closure) spaces. Whenused in the context of medical imaging,this approach admits very concise, highlevel specifications (in the order of tenlines of text) that can delineate, withhigh accuracy, the contours of aglioblastoma tumour in a 3D Magneto-Resonance scan within eight seconds,on a standard laptop. In comparison, ittakes an expert radiotherapist about halfan hour to perform this task.

The same procedure has been applied tocirca 200 cases (the well-known "BrainTumour Segmentation (BraTS) chal-lenge" dataset). Accuracy of theobtained results can be measured; thenew procedure scores among the top-ranking methods of the BraTS chal-lenge in 2017 - the state of the art in thefield, dominated by machine-learningmethods - and it is comparable tomanual delineation by human experts.Figure 1 shows the results of segmenta-tion of a tumour for one of those cases,where the top row shows the MRIimage, the middle row the result ofmanual segmentation by independentexperts, and the bottom row the resultperformed with VoxLogicA.

In the near future we plan to enhancethis work, both in the direction of clin-ical case studies and to embrace othercomputational approaches that can becoordinated and harmonised usinghigh-level logical specifications.Furthermore, the approach is very ver-satile, and its application is not limitedto a single specific type of tumour orregion in the body, paving the way forthe analysis of other types of cancer andsegmentation of various kinds of braintissue such as white and grey matter.



voxLogicA: a Spatial-Logic based tool

for Declarative Image Analysis

by Gina Belmonte (AOUS), Vincenzo Ciancia (ISTI-CNR), Diego Latella (ISTI-CNR) and Mieke Massink (ISTI-CNR)

Glioblastomas are among the most common malignant intracranial tumours. Neuroimaging protocols are

used before and after treatment to evaluate its effect and to monitor the evolution of the disease. In clinical

studies and routine treatment, magnetic resonance images (MRI) are evaluated, largely manually, and based

on qualitative criteria such as the presence of hyper-intense tissue in the image. VoxLogicA is an image

analysis tool, designed to perform tasks such as identifying brain tumours in 3D magneto-resonance scans.

The aim is to have a system that is portable, predictable and reproducible, and requires minimal computing

expertise to operate.

Figure�1:�Results�of�segmentation�of�GTV�for�TCIA�471�patient:�a)�FLAIR�acquisition�b)

Manual�segmentation�(BRATS�17�dataset)�c)�Segmentation�result�performed�with�VoxLogicA.

A recent publication introducing thetool and its application to glioblastomasegmentation can be found in [1]. Thesource code and binaries of VoxLogicAare available at the link below togetherwith a simple example of a 2D back-ground removal task, intended as a minitutorial for the tool. The theoreticalfoundations of spatial model checkingcan be found in [2] and an earlier studyon glioblastoma segmentation per-formed with the general purpose spa-tial-temporal model checker“topochecker” can be found in [3].

Link:

[L1] https://kwz.me/hyy

References:

[1] G. Belmonte, et al.:: “VoxLogicA:a Spatial Model Checker forDeclarative Image Analysis,TACAS 2019, LNCS 11427, pp.281-298, 2019. DOI: 10.1007/978-3-030-17462-0_16

[2] V. Ciancia, et al.:“Model CheckingSpatial Logics for Closure Spaces”,Logical Methods in Computer

Science Vol. 12, Nr. 4, 2016. DOI:10. 2168/LMCS-12(4:2)2016

[3] F. Banci Buonamici, et al.: “SpatialLogics and Model Checking forMedical Imaging”, STTT, Onlinefirst, 2019. DOI: 10.1007/s10009-019-00511-9

Please contact:

Vincenzo CianciaCNR-ISTI, [email protected]


Various methods have been developed torecognise visual patterns in medicalimaging [1]. Some techniques are usedfor classification of medical images; theautomatic recognition of the pathologyassociated with the given image. Othersare adopted for clustering medical imagerepositories, whose aim is to detect thedifferent pathologies characterising theimage repository. Pattern recognition isalso used for segmenting or clusteringmedical images in uniform regionswhich can correspond to high-risk areas.Finally, medical image registrationexploits pattern recognition methods forcomparing body part images captured indifferent conditions and detecting the

optimal alignment among them in orderto monitor the evolution of the disease.All these approaches are important formultiple reasons: (i) quick identifica-tion of a given disease through visuali-sation and recognition of elements tofurther investigate with accurate med-ical exams, (ii) supporting the physicianin the diagnosis process, and (iii) moni-toring the patient’s conditions overtime.

When it comes to medical image regis-tration, different methods, based mainlyon magnetic resonance (MR) images ofthe brain, have been proposed for moni-toring the temporal evolution of a

stroke. These methods have limitations,however, given that acquisition costsare high and availability of MR imagingis sometimes low. Other studies arefocused on monitoring the temporalevolution of a stroke in its acute phase.

To overcome these limitations, we pro-pose a new system based on image reg-istration techniques applied on com-puted tomography (CT) exams of thepatient's brain for monitoring the tem-poral evolution of stroke lesions [2]. Thesystem operates in two phases: (i) itevaluates past lesions which are notrelated to stroke through comparison ofpast CT exams with the most recent onerelated to stroke event; (ii) then it evalu-ates the trend of the lesion over timethrough comparison of recent CT examsrelated to the current stroke.Comparison of source and target CTexams is performed using image regis-tration with a new introduced pattern-based similarity measure in 3D. The reg-istration task aims to compute a transfor-mation function maximizing the simi-larity between the source CT exam and atransformation of the target CT exam.The similarity function is a 3D extensionof the “approximate average commonsubmatrix” measure (A-ACSM). It com-putes the similarity between two CTexams as the average volume of thelargest sub-cubes matching, to less than

New Directions for Recognizing visual Patterns

in Medical Imaging

by Alessia Amelio (University of Calabria, Italy), Lucio Amelio (independant researcher) and RadmilaJanković (Mathematical Institute of the S.A.S.A., Serbia)

New study directions are focused on the extraction and recognition of visual patterns from different

types of medical images.

Figure�1:�Flowchart�of�the�proposed�system�[2].

a few voxels, in a neighbourhood in thetwo CT exams. The proposed systemcould provide substantial support to hos-pitals as well as industrial partners bymonitoring the stroke for a period oftime that is nominated by the physician.Since the new similarity measure in theimage registration is robust to noise,comparison of CT exams is also moreaccurate than traditional registrationmethods. Figure 1 shows the flowchartof the proposed system.

In addition, we recently explored pat-tern recognition methods for clusteringdermoscopic images of different typesof skin lesions [3]. As yet, with theexception of melanoma images, theestablishment of dermoscopic imagerepositories representing differentpathologies has not been addressed inthe literature. These methods can makethe diagnosis process faster and moreaccurate, and support the design ofinteractive atlases, which can helpphysicians with a differential diagnosis.Accordingly, we exploited the 2D ver-sion of A-ACSM for clustering imagerepositories, and tested our approach ondermoscopic image databases.Specifically, A-ACSM computes thedissimilarity between two imagesstarting from the average area of thelargest square sub-matrices matching,to less than a few pixels, in the twoimages. The A-ACSM dissimilarity

measure is used in the optimisationfunction of a K-medoid-based clus-tering algorithm. The new clusteringalgorithm, called “approximate averagecommon submatrix-based K-medoids”(A-KME), is run on a dermoscopicimage repository with 12 skin diseases.Figure 2 shows the execution of the pro-posed A-KME algorithm.

At the end, we obtained very promisingresults in clustering dermoscopic imagerepositories versus competing methods.

The research directions about visualpattern recognition in the medicaldomain involved the University ofCalabria, DIMES, and theMathematical Institute of the SerbianAcademy of Sciences and Arts, Serbia.Future work will investigate the use ofA-KME algorithm on different typesof medical images. The proposedsystem will also be extensively testedon different case studies and employedin real-life contexts.

References:

[1] L. Amelio, A. Amelio:“Classification Methods in ImageAnalysis with a Special Focus onMedical Analytics”, MachineLearning Paradigms, Springer, 149,31-69, 2019.

[2] L. Amelio, A. Amelio: “CT imageregistration in acute stroke moni-toring”, MIPRO, IEEE, 1527-1532, 2018.

[3] L. Amelio, R. Janković, A. Amelio:“A New Dissimilarity Measure forClustering with Application toDermoscopic Images”, IISA, IEEE,1-8, 2018.

Please contact:

Alessia Amelio, DIMES, University ofCalabria, Rende, [email protected]



Figure�2:��Execution�of�A-

KME�clustering�algorithm�[3].

The�A-medoid�is�an�image

which�is�representative�of�a

cluster�of�images.�The

algorithm�randomly�selects�an

A-medoid�for�each�cluster�(the

number�K�of�clusters�is�an

input�parameter).�After

assigning�the�images�to�their

closest�A-medoids�in�terms�of

A-ACSM�dissimilarity

measure,�an�image�is�randomly

selected�to�swap�with�its�A-

medoid.�The�swap�is�only

performed�if�a�gain�in�the

optimisation�function�is

obtained.�The�two�last�steps�are

repeated�until�there�is�no�swap

that�can�improve�the

optimisation�function.


Solid tumours do not appear as a homo-geneous entity, but are formed by multi-clonal populations of cancer cells thatexhibit considerable spatial and tem-poral variability. Such variability canprovide valuable information on boththe aggressiveness and the probability ofresponse to therapies (therapeuticresponse) of the tumour itself [1].

Needle biopsy is a commonly used toolin clinical practice, but it is an invasiveprocedure that does not allow detectingthe entire range of potential biologicalvariation within a tumour because of thelimited number of locations wherecancer cells are sampled (incompletesampling). Clinical imaging, conversely,samples the entire tumour volume in anon-invasive way and allows pheno-typic characteristics to be extracted fromdifferent spatial and temporal levels,from macro-lesions up to the cellularand genetic scale.

However, image characteristics areoften visually evaluated and describedby radiologists or clinicians, giving riseto a subjective description of tumourimaging phenotypes, with significantintra-and inter-observer variability [2].Moreover, only about 10 % of the infor-mation contained in a digital medicalimage can usually be extracted by avisual analysis.

It was within this context that radiomicswas born; the answer to the search for aquantitative, objective and reproducibleinformation extraction method for bio-medical images. Underpinningradiomics [3] is computer vision, but italso has deep roots in statistics, whichare often underestimated. Radiomicscan be defined as the omic discipline ofclinical image analysis, whose resultsare intrinsically objective and repeat-able. An example of a radiomic pipelineis shown in Figure 1.

Radiomics converts the intrinsic infor-mation within a digital image into ahuge quantity of features (from a fewdozen up to a few thousand) that arecomputed with specific mathematicalalgorithms. The evaluated featuresdescribe imaging parameters as inten-sity, shape, size, volume, textures, etc.,related to the underlying tissue struc-tures (e.g. the neoplasm and/or the sur-rounding healthy tissues).

By using appropriate data mining tech-niques (e.g. machine learning, deeplearning), such features can be investi-gated, and the phenotypic and micro-environmental traits of a cancer tissuecan be identified, enriching the dataprovided by laboratory tests andgenomic or proteomic tests.

The identification of the “radiomic sig-nature”, in combination with otheromics data, can then be used for the

Radiomics to Support Precision Medicine

in oncology

by Sara Colantonio (ISTI-CNR), Andrea Barucci (IFAC-CNR) and Danila Germanese (ISTI-CNR)

Precision health, the future of patient care, is dependent on artificial intelligence. Of the information

contained in a digital medical image, visual analysis can only extract about 10%. Radiomics aims to

extract an enormous wealth of quantitative data from biomedical images, which could not be processed

through simple visual analysis, but is capable of providing more information on the underlying

pathophysiological phenomena and biological processes within human body. The subsequent mining of

these quantitative data can offer very useful information on the aggressiveness of the disease under

investigation, opening at the tailoring of the therapies based on a patient’s needs and at the monitoring

of the response to care. Therefore, by using specific mathematical algorithms and artificial intelligence

techniques, radiomics provides very powerful support for precision medicine, especially in oncology.

Figure�1:�The�pipeline�of�radiomic�analysis

toward�precision�medicine�(inspired�from

Fig.2�of�Nioche�et�al.,�Cancer�Res.,�78(16),

2018).�For�each�of�the�imaging�modalities

(MRI,�TC,�PET,�US),�the�first�phase�involves

a�semi-automatic�or�automatic�segmentation

of�the�relevant�regions;�this�is�followed�by�the

automatic�extraction�of�a�large�number�of

features�(descriptive�of�the�histogram,�shape,

texture�and�texture).�These�features�are�then

processed�with�machine�learning�techniques,

such�as�feature�selection�techniques�and

comparative�analysis,�to�select�the�most

robust�ones�that�correlate�to�ground�truth

biomarkers�(used�in�clinical�practice).

development of diagnostic and prog-nostic models, describing phenotypicpatterns connected to biological or clin-ical end points.

The availability of robust and validatedquantitative biomarkers is fundamentalto move precision medicine forward,and radiomics, in which imaging (andother) biomarkers are used to: (i) modelthe characteristics of each individualand the variability among individualsand (ii) predict the right treatment, forthe right patient, at the right time, epito-mises the research toward the imple-mentation of personalised approaches.

Given that cancer has an intrinsicallyhigh intra- and inter-variability, owingto a range of internal and external fac-tors, personalisation of the treatmentbecomes fundamental in oncology.

With this in mind, a multidisciplinarygroup of researchers from two CNRinstitutes, and clinicians from severalhospitals and university centres, arecombining their skills (including med-ical physics, biology, oncology, bio-medical engineering, mathematics and

computer science) to investigate theradiomic signature for the grading ofprostate cancer (PCa), as well as the useof inductive representation learningmethods.

Our pilot study investigated the associa-tion between radiomic features extractedfrom multi-parametric magnetic reso-nance imaging (mp-MRI), the ProstateImaging Reporting and Data System (PI-RADS) classification, and the tumourhistologic subtypes (using the patholo-gist Gleason score grading system), inorder to identify which of the mp-MRIderived radiomic features (signature) candistinguish high and low risk PCa,, withthe aim of integrating or replacing infor-mation obtained by solid biopsy.

Using a retrospective cohort of over 100MRI patients, radiomic features (about800) were evaluated on tumour areassegmented by the radiologists. A feed-forward method of selecting wrappertype features was used to select the fourmost relevant features. These were usedto train (10-fold cross-validation) anarrow neural network able to predictGleason score. Our method outper-

formed the majority of the worksreported in literature based on standardmachine learning techniques.

A descriptive paper was submitted toRSNA2020, another is undergoing sub-mission to BIBE2019, while furtherdata collection is underway.

References:

[1] G. Lin, K. R. Keshari, and J. MoPark: “Cancer Metabolism andTumor Heteroge-neity: ImagingPerspectives Using MR Imagingand Spectroscopy”, Contrast MediaMol Imaging 2017.

[2] S. Trattnig: “The Shift in Paradigmto Precision Medicine in Imaging:International Initiatives for thePromotion of ImagingBiomarkers”, Imaging Biomarkers,2017.

[3] S. Rizzo et al.: “Radiomics: thefacts and the challenges of imageanalysis”, European RadiologyExperimental, 2018.

Please contact:

Sara Colantonio, ISTI-CNR, Italy [email protected]



Breast cancer is the most commonly diag-nosed cancer among women worldwide.According to the latest American CancerStatistics [L1], breast cancer is the secondleading cause of death among women,and one in eight women will develop thedisease at some point in her life.

Although the incidence of breast canceris increasing, mortality from this diseaseis decreasing. This is mainly due to thebreast cancer screening programs inwhich women aged 45-74 are called tohave a mammographic exam every twoyears. Although mammography is stillthe most widely used screening method,

it suffers from two inherent limitations:a low sensitivity (cancer detection rate)in women with dense breastparenchyma, and a low specificity,causing unnecessary recalls. The lowsensitivity in women with dense breastsis caused by a “masking effect” of over-lying breast parenchyma. Furthermore,the summation of normal breastparenchyma on the conventional mam-mography may occasionally simulate acancer. In recent years, new imagingtechniques have been developed:tomosynthesis, which can produce 3Dand 2D synthetic images of the breast,new MRI techniques with contrast

medium and breast CT. However,thanks to screening programs,numerous mammographic images canbe collected from hospitals to buildlarge datasets on which it is possible toexplore AI techniques.

In recent years, new methods for imageanalysis have been developed. In 2012,for the first time, ImageNet Large ScaleVisual Recognition Competition(ILSVRC), the most important imageclassification challenge worldwide, waswon by a deep learning-based classifiernamed AlexNet [1]. Starting from thisresult, the success of deep learning on

Deep-Learning based Analyses of Mammograms

to Improve the Estimation of breast Cancer Risk

by Francesca Lizzi (National Institute for Nuclear Physics, Scuola Normale Superiore, National ResearchCouncil, University of Pisa), Maria Evelina Fantacci (National Institute for Nuclear Physics, University of Pisa)and P. Oliva (National Institute for Nuclear Physics, University of Sassari)

Breast cancer is the most commonly diagnosed cancer among women worldwide. Survival rates strongly

depend on early diagnosis, and for this reason mammographic screening is performed in developed

countries. New artificial intelligence-based techniques have the potential to include and quantify

fibroglandular (or dense) parenchyma in breast cancer risk models.


visual perception problems is inspiringmuch scientific work, not only on nat-ural images, but on medical images too[2]. Deep learning-based techniqueshave the advantage of very high accu-racy and predictive power at theexpense of their interpretability.Furthermore, they usually need a hugeamount of data and a large computa-tional power to be trained.

At the Italian National Institute forNuclear Physics (INFN), within theframework of a PhD in data science[L2] of Scuola Normale Superiore ofPisa, University of Pisa and the ISTI-CNR, we are working to apply deeplearning models to find new image bio-markers extracted from screening mam-mograms that can help with early diag-nosis of breast cancer.

Previously [3], we trained and evalu-ated a breast parenchyma classifier inthe BI-RADS standard, which is madeof four qualitative density classes(Figure 1), using a deep convolutionalneural network and we obtained verygood results compared to other work.Our research activities are continuingwith a larger dataset and more ambi-tious objectives. We are collecting datafrom Tuscany screening programs andthe ever-expanding dataset currentlyincludes:

• 2,000 mammographic exams (8,000images, four per subject) of healthywomen labelled by the amount offibroglandular tissue. These examshave been extracted from the Hospi-tal of Pisa database.

• 500 screen-detected cases of cancer,90 interval cancer cases and 270 con-trol exams along with the histologicreports and a questionnaire with theknown breast cancer risk factors,such as parity, height, weight andfamily history. It is possible to accessall the mammograms prior to diagno-sis for each woman. These examshave been extracted from the North-West Tuscany screening database.

The goal of our work is multi-fold andmay be summarised as follows: • to explore the robustness of deep

learning algorithms with respect tothe use of different mammographicsystems, which usually result in dif-ferent imaging properties.

• to define a deep learning model ableto recognise the kind and nature ofthe malignant masses depicted inmammographic data based on therelated histologic reports.

• to investigate the inclusion of thefibroglandular parenchyma in breastcancer risk models in order toincrease the predictive power of cur-rent risk prediction models. In thisrespect, changes in dense parenchy-ma will be monitored over time

through image registration tech-niques, to understand how its varia-tion may influence cancer risk. Fur-thermore, we will investigate the roleof dense tissue in the onset of intervalcancers and the correlation amongboth local and global fibroglandulartissue and other known risk factors soas to quantify the risk in developing abreast cancer.

Links:

[L1] https://kwz.me/hy5 [L2] https://datasciencephd.eu/

References:

[1] Alex Krizhevsky et al.: “ImageNetclassification with deepconvolutional neural networks”,NIPS'12 Proceedings, 2012.

[2] Geert Litjens et al.: “A survey ondeep learning in medical imageanalysis”, Medical Image Analysis,2017.

[3] F. Lizzi et al.: “ResidualConvolutional Neural Network forbreast density classification”,BIOINFORMATICS Proceedings,2019, ISBN: 978-989-758-353-7.

Please contact:

Francesca Lizzi, National Institute forNuclear Physics, Scuola NormaleSuperiore, National Research Council,University of Pisa, [email protected]

Figure�1:�The�four�density�classes�are�shown�as�reported�in�the�BI-RADS�Atlas.�The�classes�are�defined�through

textual�description�and�examples�and�are�named�A,�B,�C�and�D�in�order�of�increasing�density.

Data from medical 3D sensors, such ascomputer tomography (CT) and mag-netic resonance imaging (MRI), give 3Dinformation as output, thereby creatingthe opportunity to model 3D objects(e.g., organs, tissues, lesions) existinginside the body. These quantitativeimaging techniques play a major role inearly diagnosis and make it possible tocontinuously monitor the patient. Withthe improvement of these sensors, alarge amount of 3D data with high spa-tial resolution is acquired. Developingefficient processing methods for thisdiverse output is essential.

Our “Content Based Analysis ofMedical Image Data” project, con-ducted with Pázmány Péter CatholicUniversity, Faculty of InformationTechnology and Bionics (PPKE ITK)[L1], concentrated on the developmentof image processing algorithms for mul-timodal medical sensors (CT and MRI),applying content-based information,saliency models and fusing them withlearning-based techniques. We devel-oped fusion methods for efficient seg-mentation of medical data, by inte-grating the advantages of generativesegmentation models, applying tradi-tional, “handcrafted” features; and thecurrently preferred discriminativemodels using convolutional features.By fusing the two approaches, the draw-backs of the different models can bereduced, providing a robust perform-ance on heterogeneous data, even withpreviously unseen data acquired by dif-ferent scanners.

The fusion model [1] was introducedand evaluated for brain tumour seg-mentation on MRI volumes, using anovel combination of multiple MRImodalities and previously built healthytemplates as a first step to highlightpossible lesions. In the generative partof the proposed model, a colour- andspatial-based saliency model wasapplied, integrating a priori knowledge

on tumours and 3D informationbetween neighbouring scan slices. Thesaliency-based output is then com-bined with convolutional neural net-works to reduce the networks’ even-tual overfitting which may result inweaker predictions for unseen cases.By introducing a proof-of-conceptmethod for the fusion of deep learningtechniques with saliency-based, hand-crafted feature models, the fusionapproach has good abstraction skills,yet can handle diverse cases for whichthe net was less trained.

In a similar manner, we also imple-mented a technique for liver segmenta-

tion in CT scans. First, a pre-processingwas introduced using a bone mask tofilter the abdominal region (this isimportant in the case of whole-bodyscans). Then a combination of regiongrowing, and active contour methodswas applied for liver region segmenta-tion. This traditional feature-basedtechnique was fused with a convolu-tional neural network’s predictionmask to increase segmentation accu-racy (Figure 1).

The proposed techniques [2] have beensuccessfully applied in the “zMed”project [L2], a four-year project run byZinemath Zrt., the Machine Perception



Content-based Analysis of Medical Image Data

for Augmented Reality based Health Applications

by Andrea Manno-Kovacs (MTA SZTAKI / PPKE ITK), Csaba Benedek (MTA SZTAKI) and Levente Kovács (MTA SZTAKI)

Novel 3D sensors and augmented reality-based visualisation technology are being integrated for innovative

healthcare applications to improve the diagnostic process, strengthen the doctor-patient relationship and open new

horizons in medical education. Our aim is to help doctors and patients explain and visualise medical status using

computer vision and augmented reality.

Figure�1:�Liver�segmentation�and�3D�modelling�on�CT�data:�Segmentation�result�in�axial,

coronal,�sagittal�view�and�the�rendered�3D�model�of�the�liver.

Figure�2:�The�main�motivations�of�the�zMed�project�[L2].

Research Laboratory in the Institute forComputer Science and Control of theHungarian Academy of Sciences (MTASZTAKI) [L3] together with theUniversity of Pécs (PTE). TheDepartment of Radiology at PTE pro-vides input data of different modalities(CT and MRI) for the segmentation algo-rithms, developed by the computervision researchers of MTA SZTAKI.Based on these tools, Zinemath Zrt isdeveloping a software package, whichprovides 3D imaging and visualisationtechnologies for unified visualisation ofmedical data and various sorts of spatialmeasurements in an augmented realitysystem. By creating a completely novelvisualisation mode and exceeding thecurrent display limits, the softwarepackage applies novel technologies, suchas machine learning, augmented realityand 3D image processing approaches.

The developed software package isplanned to be adaptable to multiplemedical fields: medical education andtraining for future physicians, intro-ducing the latest methods moreactively; improving the doctor-patientrelationship by providing explanationsand visualisations of the illness; sur-gical planning and preparation in thepre-operative phase to reduce the plan-ning time and contributing to a moreprecisely designed procedure (Figure2).

This work was supported by the ÚNKP-18-4-PPKE-132 New NationalExcellence Program of the HungarianGovernment, Ministry of HumanCapacities, and the HungarianGovernment, Ministry for NationalEconomy (NGM), under grant numberGINOP-2.2.1-15-2017-00083.

Links:

[L1]: https://itk.ppke.hu/en[L2]: http://zinemath.com/zmed/[L3]: http://mplab.sztaki.hu

References:

[1] P. Takacs and A. Manno-Kovacs:“MRI Brain Tumor SegmentationCombining Saliency andConvolutional Network Features”,Proc. of CBMI, 2018.

[2] A. Kriston, et al.: “Segmentation ofmultiple organs in ComputedTomography and MagneticResonance Imaging measurements”,4th International Interdisciplinary3D Conference, 2018.

Please contact:

Andrea Manno-KovacsMTA SZTAKI, [email protected]


Identifying disease-causing geneticcharacteristics starts with analysingdatasets containing the genetic informa-tion of both healthy individuals andpatients with a disease of interest. Thedata analysis provides direction to dis-ease experts and lab researchers, whocan experimentally test whether agenetic variant indeed causes disease.Validated disease-causing genetic vari-ants provide insight into the cellularprocesses involved in disease, which isthe starting point for drug development.

Today’s predominant technique foranalysing genome datasets, calledgenome-wide association studies(GWAS), ensures that cause (geneticvariant) and effect (disease) can belinked in a way the human mind cangrasp. GWAS examine each individualgenetic variants for correlation with dis-ease, following well-understood statis-tical principles. GWAS allows theresearcher to easily interpret findings

and has been very successful: manypotentially disease-causing variantshave been detected for various diseases.

However, several diseases stubbornlyresist such “human intelligence-basedapproaches”, as their genetic architec-ture is difficult to unravel. One architec-tural feature that complicates analysesconsiderably is epistasis[1]: geneticvariants do not necessarily just add uptheir effects to establish effects, butoperate in terms of logical combina-tions. Consider, for example, three vari-ants A, B and C, which establish the dis-ease-causing effects if (and only if) A isnot there, or B and C are both there.Such complex logical relationshipsreflect common biochemical gateways.

Analysing diseases with a moreinvolved genetic architecture, such ascancer, type II diabetes or ALS, in termsof “human mind perceivable”approaches clearly has reached certain

limits. So, an immediate question is: ifthe human mind is struggling, can AIhelp out?

This motivated CWI researchersMarleen Balvert and AlexanderSchönhuth to develop new, AI-basedtechniques for identifying complexcombinations of genetic characteristicsthat are associated with disease. Thechallenge is twofold.

First, genome datasets contain millionsof genetic variants for thousands or tensof thousands of individuals. Deepneural networks - currently establishedamong the most successful classifica-tion techniques [2] - offer enhancedopportunities in processing largedatasets. This motivated Balvert andSchönhuth to employ deep neural net-works.

Second, deep neural networks havebeen predominantly developed for

Artificial Intelligence: understanding Diseases

that People Cannot understand?

by Marleen Balvert and Alexander Schönhuth (CWI)

Many diseases that we cannot currently cure, such as cancer, Alzheimer’s and amyotrophic lateral

sclerosis (ALS), are caused by variations in the DNA sequence. It is often unknown which characteristics

caused the disease. Knowing these would greatly help our understanding of the underlying disease

mechanisms, and would boost drug development. At CWI we develop methods based on artificial

intelligence (AI) to help find the genetic causes of disease, with promising first results.

image classification. Unlike image data- the structure of which can be graspedimmediately - genetic data has a struc-ture that is governed by the laws of evo-lution and reproduction. Arranginggenetics data to act as input to deepneural networks therefore requiresexpert knowledge.

Together with ALS expert Jan Veldinkfrom UMC Utrecht, Balvert andSchönhuth took on the challenge ofdeveloping a deep neural network toclassify healthy individuals from ALSpatients using data from over 11,000people. The data were collected throughProject MinE, a global genome dataproject that deals with ALS. Note thatthe CWI researchers were guided by theidea to design a general neural networkarchitecture for diseases with a complexgenetic architecture, so as to not neces-sarily specialise in a particular disease.

The team implemented a two-step pro-cedure [3]. First, a relatively light-weight neural network identifies pro-moter regions - parts of the genome thatinitiate the reading of a gene - that areindicative of disease. Upon identifyingseveral tens out of the 20,000 promoterregions an ultra-deep neural networkpredicts whether someone is affected byALS based on the variants captured bythe selected promoter regions.

If the neural network achieves goodpredictive performance, it has“learned” how to identify disease. Thegenetic architecture of the disease isthus captured by the wirings of theneural network.

Balvert, Schönhuth and their team wereintrigued and enthusiastic to observe thattheir networks achieved excellent per-formance in predicting ALS; ALS hasbeen marked as a disease whose geneticarchitecture is most difficult to disen-tangle. The networks achieved 76 % pre-diction accuracy, surpassing the simpler,“human mind perceivable” approachesthat achieved 64 % accuracy at best.Further improvements are still possible.

These highly encouraging results pointout that AI can do an excellent job inunderstanding complex genetic disor-ders. However, we will encounter manyfurther issues before AI will find its wayinto clinical practice. Most importantly,while AI can understand the geneticarchitecture of a disease, we are not ableto fully disentangle the wirings a neuralnetwork uses for its predictions, and thehuman mind still has not been helped.

But there is hope: method developmentthat aims at human understanding of AIis one of the most active areas ofresearch of our times.

References:

[1] V. Tam, et al.: “Benefits and limita-tions of genome-wide associationstudies”, Nature Review Genetics,2019.

[2] J. Schmidhuber: "Deep learning inneural networks: An overview",Neural networks 61, 2015: 85-117.

[3] B. Yin, et al.: “Using the structureof genome data in the design ofdeep neural networks for predictingamyotrophic lateral sclerosis fromgenotype”, Bioinformatics (Proc.of ISMB/ECCB 2019), to appear.

Please contact:

Marleen Balvert, CWI, The [email protected]



CWI�researchers�are�currently�developing�AI�techniques�to�help�identify�the�genetic

characteristics�that�lead�to�disease.�Picture:�Shutterstock.

Viruses, such as HIV, Ebola and Zika,populate their hosts as a viral quasi-species: a collection of geneticallyrelated mutant strains, which rapidlyevolve by the accumulation of ever more

mutations as well as recombinationamong the strains. To determine theright treatment for infected people, it iscrucial to draw a clear picture of thevirus DNA that affects the patients [1].

The genome of an HIV strain, forexample, consists of approximately10,000 letters. While most virus strainsgenerally share most letters, compara-tively rare, but utterly relevant differ-

the Genetic Diversity of viruses on a Graphical

Map: Discovery of Resistant and virulent Strains

by Alexander Schönhuth (CWI and Utrecht University) and Leen Stougie (CWI and VU Amsterdam)

Many life-threatening viruses populate their hosts with a cocktail of different strains, which may mutate

insanely fast, protecting the virus from human immune response or medical treatment. Researchers at

CWI have designed a method, named Virus-VariationGraph (Virus-VG) [3], that puts all strains onto a

graphical map, which facilitates more reliable and convenient identification of potentially resistance-

inducing or particularly lethal strains.


ences can decisively determine theirclinically relevant properties, such asresistance to treatment, or their viru-lence. To draw a clear picture, it is nec-essary to, first, reconstruct the genomesof the different strains at full length, andsecond, to estimate the relative propor-tions of the strains that make up theviral quasispecies, the mix of strainsaffecting an individual patient.

Applying modern sequencing tech-niques to virus DNA extracted frominfected people yields millions ofsequence fragments, however, and notfull-length genomes of strains. The taskis now to assign the (many) fragmentsto different strains. Each genome thenneeds to be reconstructed at full length,and its relative abundance estimatedwithin the mix of strain genomes. Thisprocedure is commonly referred to asviral quasispecies assembly. It is impor-tant to note that virus referencegenomes, which seem to promise orien-tation during the assembly process canconsiderably disturb this procedure, byintroducing biases that can decisivelyhamper the assembly.

Viral quasispecies assembly is verychallenging, particularly in the absenceof reference genomes, and is not yet afully resolved issue. Schönhuth, Stougieand their co-workers have recentlytaken big strides in this area.

Their idea was to put all fragments (orbetter: contigs, which are contiguouspatches of fragments that together muststem from an identical strain; these canbe reliably determined using othermethods [2]) on a directed, graphicalmap. In such a map, full-length pathscorrespond to full-length genomes.Further, the relative abundance of astrain genome then relates to the relativenumber of fragments that make part ofthe path through this map. This graph-ical map then allows low-frequencystrains - paths through the map that aresupported by rather low amounts offragments - to be conveniently high-lighted. The identification of low-fre-quency strains is important in theanalysis of viral quasispecies. When notsubjected to a careful analysis, low-fre-quency strains tend to be neglected, andconsequently such strains may induceresistance to treatment or emerge as par-ticularly virulent after treatment.

Schönhuth, Stougie and co-workershave developed a method, Virus-

VariationGraph (Virus-VG) that imple-ments these ideas. This was achievedthrough the construction of “variationgraphs” from the input fragments(which are contigs, see above).Variation graphs have become popularrecently in the analysis of genomes. Thegeneral idea is to transform a collectionof related genomes into a variationgraph, which allows for types ofgenome analyses that were hithertounconceivable. Usually, however, vari-ation graphs are constructed from full-length genomes, which prevents the useof variation graphs for viral quasi-species assembly.

Here, Schönhuth, Stougie and co-workers generalised the concept of vari-ation graphs, which allowed them to beflexibly constructed from shortersequence patches. They designed anoptimisation problem whose solutionconsists of laying out the paths that cor-respond to strain genomes, and assignsrelative abundances to those paths. SeeFigure 1 for an illustration of the steps.

They were able to demonstrate theadvantages of the new graph-basedapproach over other viral quasispeciesapproaches (all of which use referencegenomes), in various relevant aspects,such as strain coverage, length ofgenomes, and abundance estimates. Thismethod seems especially beneficial foridentifying low-frequency strains,which is of particular interest for theabove-mentioned clinical reasons.

Overall, Schönhuth, Stougie and co-workers succeeded in providing the firstsolution to the viral quasispeciesassembly problem that does not onlyyield the genomes of the strains at max-imal length, but also reliably estimatestheir relative abundances, withoutmaking use of existing referencegenomes. Virus-VG is publicly avail-able at [L1].

Link:

[L1] https://kwz.me/hy3

References:

[1] S. Posada-Cespedes, D. Seifert andN. Beerenwinkel: “Recentadvances in inferring viral diversityfrom high-throughput sequencingdata”, Virus research, 239, 17-32,2017.

[2] J. Baaijens, et al.: “Full-length denovo viral quasispecies assemblythrough variation graphconstruction”, Bioinformatics,btz443,https://doi.org/10.1093/bioinformatics/btz443, 2019

[3] J. Baaijens, A.Z. El Aabidine, E.Rivals, A. Schönhuth: "De novoassembly of viral quasispeciesusing overlap graphs", GenomeResearch, 27(5), 835-848, 2017.

Please contact:

Alexander Schönhuth.CWI, [email protected]

Figure�1:�At�CWI,�researchers�have�developed�Virus-VG,�an�algorithm�that�is�more�reliable�and

convenient�to�use�for�assembling�viral�quasispecies�than�earlier�methods.�Picture:�CWI.

To reduce tissue consumption when per-forming these tests, our team imple-mented a variation of IHC that wetermed micro-IHC (µIHC). The rationalewas to reduce the area stained on thetissue section only to a region of interestat the micrometre scale, for example asmall portion of the tumour. To performµIHC we use a microfluidic probe(MFP), a liquid scanning probe, which isa device that locally deposits chemicalson specific regions of the tissue at themicrometre-length scale. This way, weperform not only local IHC staining butalso can use several biomarkers on thesame tissue section [1].

Using the capabilities of µIHC, we envi-sioned its application in optimising anti-body-antigen reaction, a critical step inquality control of the IHC test. Currently,new antibodies must be tested across arange of concentrations and incubationtimes with multiple tissue sections to testtheir specificity and sensitivity. This canconsume a great number of valuablesamples while providing only limitedinformation. The high tissue consump-tion reduces the feasibility of optimisingeach batch of antibodies, although it isknown that they can present variations inperformance. These variations are ampli-fied when more than one tissue section isused, which often come from differentsources or are prepared differently.

In these circumstances, the use of µIHCon a tissue section can generate an arraywith several conditions, limiting tissueconsumption and equalising the wholedownstream process. This potentiallyreduces the variability that is inherentwhen using different samples.

Defining “optimal” for a stainWith this methodology in hand we werefaced with a rather tricky question: whatis an “optimal” stain? We realised thereis no straightforward answer, since the“optimal” stain varies depending on thetissue type. Take for example a commonbiomarker in breast cancer, HER2. Theoptimal stain in a healthy tissue wouldbe “no stain”. Any stain that we observeis qualified as a false positive. However,in about 20 % of cases of breast cancer,the biomarker is present, producing astain of varying intensities. “No stain”in this circumstance is regarded as afalse negative, but even changes in stainintensity could misdirect the treatmentprovided by clinicians. Other stainingartefacts, such as overstaining, canmake the interpretation of the test morecomplex by masking the signal.

Using machine learning to obtain anoptimal stain qualificationWe asked several experts to classify ourstains in “good”, “acceptable” and “notacceptable”. Nevertheless, we knew

that manual labour was not an efficientand scalable way to perform this (orany) optimisation. Therefore, wedecided to use the capabilities ofmachine learning to generate a moreobjective way to score the tissues, usingthe references provided by the patholo-gists [2].

We imaged all tested conditions andproceeded to extract features based onintensity, texture and the Fourier trans-form. These features were used for twopurposes. On one side, the algorithmhad to understand what kind of tissuewas being considered. As mentioned,the expected staining is not the same ifwe are analysing a healthy tissue over atumour tissue, making tissue identifica-tion an important facet. To do so, wetrain a classifier method that can learnfrom labelled data, a Support VectorMachine, with sets of features until weidentify a set that gives the best separa-tion between the pre-defined classes.Once this is done, the images that werenot used in the training are analysed andthe algorithm predicts their probabilityof belonging to a certain tissue type.

On the other side, the algorithmanalyses the contrast level between thedifferent compartments observed in thecell by looking at the intensities of eachregion. This is necessary to understand



Figure�1:�Schematic�workflow

for�the�processing�of�a�tissue�to

obtain�the�quality�value�(QV)

parameter�manifold.

Improved Antibody optimisation for tumour

Analysis through the Combination of Machine

Learning with New Molecular Assay

by Anna Fomitcheva Khartchenko (ETH Zurich, IBM Research – Zurich), Aditya Kashyap and Govind V.Kaigala (IBM Research – Zurich)

The role of a pathologist is critical to the cancer diagnosis workflow: they need to understand patient

pathology and provide clinicians with insights through result interpretation. To do so, pathologists and

their laboratory teams perform various investigations (assays) on a biopsy tissue. One of the most

common tests is immunohistochemistry (IHC), which probes the expression levels for certain proteins

that characterise the tissue, called biomarkers. This test enables sub-classification of the disease and

is critical for the selection of a treatment modality. However, the number of biomarkers is constantly

increasing, while the size of the biopsy is reducing due to early testing and more sensitive methods.

whether the stain, for the particulartissue type we are analysing, is astaining artefact. With all parts in place,the contrast level and the tissue typeidentifier are combined into an indi-cator, the “Quality Value” (QV), whichgives the probability of a good qualitystain ranging from 1 (best) to 0 (worst)depending on the tissue type (see Figure1). We extracted the QVs from ouranalysed conditions and produced amanifold that provides essential infor-mation for optimal staining of a tissuefor the patient.

OutlookThe information obtained from thisstudy can be applied on biopsy samples

of individual patients to find the beststaining conditions without consumingmuch of the sample. The remainder ofthe biopsy can then be used for otherdiagnostic tests. We believe that in thisway the number of errors - defined asfalse positives and false negatives -caused by inadequate staining condi-tions may be reduced, however we areyet to decisively prove this. We hope todemonstrate such validation in futurework.

We also believe the convergence ofmachine learning approaches withimage processing and new implementa-tions for performing biochemical assayson tissue sections will together lead to

more accurate tumour profiling andthereby a more reliable diagnosis.

References:

[1] R. D. Lovchik, et al.: “Micro-immunohistochemistry using amicrofluidic probe. Lab Chip 12”,1040-1043 (2012).

[2] N. M. Arar, et al.: “High-qualityimmunohistochemical stains throughcomputational assay parameteroptimization”, IEEE Trans. Biomed.Eng. pp, 1–1 (2019).

Please contact:

Govind V. KaigalaIBM Research – Zurich, [email protected]


Only 10–14 % of drug candidatesentering clinical trials actually reach themarket as medicine, with an estimatedUS $2–3 billion price tag for each newtreatment [1]. Despite enormous scien-tific and technological advances inrecent years, serendipity still plays amajor role in anticancer drug discoverywithout a systematic way to accumulateand leverage years of R&D to achievehigher success rates in the process. Atthe moment, a drug is usually designedby considering which protein targetmight induce signalling pathway cas-cades lethal for tumour cells. After thisinitial design phase, the efficacy of acompound on specific tumour typesrequires intensive experimental valida-tion on cell lines. The costs of this exper-imental phase can be prohibitive and anysolution that helps to decrease thenumber of required experimental assayscan provide an incredible competitiveadvantage and reduce time to market.

In this context, IBM Research devel-oped PaccMann [2,3], an in-silico plat-form for compound screening based onthe most recent advances in AI for com-putational biochemistry. The modeldeveloped implements a holistic multi-

modal approach to drug sensitivitycombining three key data modalities:anticancer compound structure in theform of SMILES, molecular profile ofcell lines in the form of gene expressiondata and prior knowledge in the form ofbiomolecular interactions. PaccMannpredicts drug sensitivity (IC50) ondrug-cell-pairs while highlighting themost informative genes and compoundsub-structures using a novel contextualattention mechanism. Attention mecha-nisms have gained popularity in recentyears, since they enable interpretablepredictions by using specific layers thatallow the model to focus and assignhigh attention weights to input featuresimportant for the task of interest.

PaccMann has been trained and vali-dated on GDSC [L1], a public dataset ofcell lines screened with a collection ofcompounds. The method outperforms abaseline based on molecular finger-prints and a wide selection of deeplearning-based techniques in an exten-sive cross-validation benchmark.Specifically, PaccMann achieves highprediction performance (R2 = 0.86 andRMSE = 0.89, see Figure 1), outper-forming previously reported state-of-

the-art results for multimodal drug sen-sitivity prediction.

To showcase the explainability ofPaccMann, its predictions on a ChronicMyelogenous Leukaemia (CML) cellline for two extremely similar anti-cancer compounds (Imatinib andMasitinib) have been analysed. Theattention weights of the molecules aredrastically different for the compounds’functional groups whereas theremaining regions are unaffected (seeFigure 2, top). The localised discrep-ancy in attention centred at the differentrings suggests that these substructuresdrive the sensitivity prediction for thetwo compounds on the CML cell line.On the gene attention level (see Figure2, bottom) a set of genes has beendetected as relevant. Interestingly, theDDR1 protein is a member of ReceptorTyrosine Kinases (RTKs), the samegroup of cell membrane receptors thatboth considered drugs inhibit. DDR1 aswell as the other highlighted genes havebeen previously reported in cancer liter-ature, especially in leukaemia. Thesefindings indicate that the genes thatwere given the highest attention weights

AI Enables Explainable Drug Sensitivity Screenings

by Matteo Manica, Ali Oskooei, and Jannis Born (IBM Research)

Accelerating anticancer drug discovery is pivotal in improving therapies and patient prognosis in cancer

medicine. Over the years, in-silico screening has greatly helped enhance the efficiency of the drug discovery

process. Despite the advances in the field, there remains a need for explainable predictive models that can

shed light onto the anticancer drug sensitivity problem. A team of scientists at the Computational Systems

Biology group within IBM Research has now proposed a novel AI approach to bridge this gap.

are indeed crucial players in the pro-gression and treatment of leukaemia.

To quantify the drug attention on a largerscale, a collection of screened drug-cellline pairs has been considered. For eachdrug, the pairwise distance matrix of allattention profiles was computed.Correlating the Frobenius distance ofthese matrices for each pair of drugswith their Tanimoto similarity (estab-lished index for evaluating drug simi-

larity based on fingerprints) revealed aPearson coefficient of 0.64 (p < 1e-50).The fact that the attention similarity ofany two drugs is highly correlated withtheir structural similarity demonstratesthe model's ability to learn insights oncompounds’ structural properties. As aglobal analysis of the gene attentionmechanism, a set of highly attendedgenes has been compiled by analysingall the cell lines. Pathway enrichmentanalysis [L2] on this set identified a sig-

nificant activation (adjusted p<0.004) ofthe apoptosis signalling pathway [L3].IC50 prediction is in essence connectedto apoptosis (cell death) and the atten-tion analysis suggests that the model isfocused on genes connected to thisprocess, thus confirming the validity ofthe attention mechanism.

PaccMann paves the way for futuredirections such as: drug repositioningapplications as it enables drug sensi-tivity prediction for any given drug-cellline pair, or leveraging the model incombination with recent advances insmall molecule generation using genera-tive models and reinforcement learningto design novel disease-specific, or evenpatient-specific compounds. This opensup a scenario where personalised treat-ments and therapies can become a con-crete option for patient care in cancerprecision medicine.

An open source release of PaccMannand the related codebase can beaccessed on GitHub [L4]. A version ofthe model has been trained on publiclyavailable data for the prediction of drugsensitivity (IC50) and has beendeployed on IBM Cloud [L5]. Themodel predicts drug response on a set of970 cell lines generated from multiplecancer types given a compound inSMILES format.

Links:

[L1] https://www.cancerrxgene.org/[L2] http://amp.pharm.mssm.edu/Enrichr/[L3] http://pantherdb.org[L4] https://kwz.me/hyS[L5]: https://ibm.biz/paccmann-aas

References:

[1] G. Schneider: “Mind and Machinein Drug Design”, Nature MachineIntelligence.

[2] A. Oskooei, J. Born, M. Manica etal.: “PaccMann: Prediction of anti-cancer compound sensitivity withmulti-modal attention-based neuralnetworks”, WMLMM, NeurIPS.

[3] M. Manica, A. Oskooei, J. Born etal.: “Towards Explainable Anti-cancer Compound Sensitivity Pre-diction via Multimodal Attention-based Convolutional Encoders”,WCB, ICML.

Please contact:

Matteo ManicaIBM Research, [email protected]



Figure�2:�Neural�attention�on�molecules�and�genes.�Top:�The�molecular�attention�maps�on�the

top�demonstrate�how�the�model’s�attention�is�shifted�when�the�Thiazole�group�(Masitinib,�left)�is

replaced�by�a�Piperazine�group�(Imatinib,�right).�The�change�in�attention�across�the�two

molecules�is�particularly�concentrated�around�the�affected�rings,�signifying�that�these�functional

groups�play�an�important�role�in�the�mechanism�of�action�for�these�Tyrosine-Kinase�inhibitors

when�they�act�on�a�CML�cell�line.��Bottom:�The�gene�attention�plot�depicts�the�most�attended

genes�of�the�CML�cell�line,�all�of�which�can�be�linked�to�leukemia.

Figure�1:�Performance�of�PaccMann

on�unseen�drug-cell�line�pairs.

Scatter�plot�of�the�correlation

between�true�and�predicted�drug

sensitivity�by�a�late-fusion�model

ensemble�on�the�cross-validation

folds.�RMSE�refers�to�the�plotted

range,�since�the�model�was�fitted�in

log�space.


A clinical pathway is a multi-discipli-nary view of the care process for a groupof similar patients suffering from thesame disease and with predictable evo-lution [1]. While scheduling the care fora single patient is straightforward,scheduling the care for a group ofpatients, and under limited resources ismuch more complex. Operating clinicalpathways is very challenging because ofthe need to achieve the timely deliveryof treatments like chemotherapy. It alsorequires a thorough knowledge of thehistory of each patient. Typical carequality indicators monitor the deviationfrom the ideal pathway and studies havestressed their correlation with cancersurvival rate. Moreover, the actualworkflow might be quite different fromthe defined workflow owing to adapta-tions resulting from a multidisciplinarycontext and the high level of personali-sation.

Within the PIPAS project [L1], togetherwith the Université catholique deLouvain, we have been engineering sucha pathway. We started by developing atoolbox supporting the modelling andoptimal scheduling of complex work-flows [2]. To engineer medical processmodels, we defined and implementeddifferent operators to combine/distin-guish specific treatment for multi-pathology patients and to provide globalor more focused viewpoints for specificagents (e.g. patient, nurse radiotherapist)or clinical departments.

Workflows of this kind need to beenacted precisely in order to ensurehigh quality of care to the pool ofpatients given the available resources(e.g. staff, beds, drug stocks). In orderto orchestrate the work of a oncologyday hospital, we developed an onlinescheduler dealing with the appoint-ments and global nurse and room allo-cation [3]. Based on all the inputs illus-trated in Figure 1, our tool provides

prescriptive analytics capabilities tothe nurse in charge of setting appoint-ments, i.e. it will help identify the nextappointment date by looking at thetime window allowed by the patientcare indicator. Il will also minimise

impacts on other patients by using aglobal view on the whole pool ofknown or even expected patients overtheir whole treatment period. Eachdeviation, such as a partial treatment orno show, is also immediately consid-ered and recomputed using an efficientconstraint local search engine calledOscaR.CBLS [L2]. The scheduler alsoensures a smoother repartition of the

workload and triggers an alarm ifinsufficient resources are provisioned.Figure 2 shows how the scheduler canmaintain a high care quality indicator(>90 %) until the service load becomesunmanageable.

The fact that the actual and theoreticalworkflows differ from one another,impacts the relevance of the proposedguidance. To address this risk, a recon-ciliation process analyses the path cap-tured in patient health records to detectthe presence of extra transitions,measure their relative frequencies andassess the global variability amongpatients. Predictive data analytics such

Combining Predictive and Prescriptive Analytics

to Improve Clinical Pathways

by Christophe Ponsard and Renaud De Landtsheer (CETIC)

Within a medical setting, clinical pathways enable efficient organisation of care processes, which

benefit both the patient and hospital management. Digital health analytics plays a critical role in the

successful deployment of clinical pathways. Two key aspects learned from our experience are the

engineering of accurate workflow models and the use of online schedule optimisation, enforcing

both care and resource constraints.

Figure�1:��Prescriptive�appointment�scheduler.

Figure�2:�Simulation�of�service�behaviour�with�increasing�patient�load�until�service�saturation.

as data mining are used here. More spe-cific techniques such as process miningcan even help in inferring workflowsfrom data. The result can then be usedeither to feed the tooling with a moreaccurate model or to reshape the organi-sation to enforce the wished workflow.This is our current progress and our nextstep is to start collecting and analysingdata from our new Institute of Analyticsfor Health (INAH) in charge of pro-viding an ethically controlled access toelectronic health information based onthe Walloon Health Network.

Links:

[L1] https://www.cetic.be/PIPAS-1169 [L2] https://bitbucket.org/oscarlib/oscar[L3] https://www.cetic.be/INAH

References:

[1] H. Campbell and al.: “Integratedcare pathways”, British MedicalJournal pp. 133{137 (1998).

[2] C. Damas, B. Lambeau, A. vanLamsweerde: “Transformationoperators for easier engineering ofmedical process models”,SEHC@ICSE, 2013.

[3] C. Ponsard and al.: “Quality ofCare Driven Scheduling of ClinicalPathways Under Resource andEthical Constraints”, ICEIS(Revised Selected Papers) 2017:162-186.

Please contact:

Christophe Ponsard, CETIC, Belgium+32 472 56 90 [email protected]



Clinical decision making is often fraughtwith difficulties related to extracting theright information from huge amounts ofdiverse data. Complex diseases are dis-eases in which diagnosis making is diffi-cult. This may be, for instance, becausesymptoms are non-specific (differentdiseases cause similar symptoms), orthere is high variability between individ-uals in how the disease manifests itself,or there is no objective gold standardregarding diagnosis.

An example of a complex disease isdementia. It is important to detectdementia as early as possible, beforelate-stage symptoms, such as severememory problems become obvious, andthere is no longer any room to improvequality of life. It is also important to dis-cern between the different forms ofdementia (differential diagnosis), inorder to provide appropriate interven-tions. Different forms of dementiainclude Alzheimer’s disease (the mostprominent form of dementia), vasculardementia, dementia with Lewy bodies,and frontotemporal dementia. To con-fuse the matter, healthy subjects maypresent with memory complaints similarto those of dementia patients.

Clinicians need to combine a wide rangeof data sources upon which to base deci-

sions. This information may range fromimaging data (MRI, CT, and sometimesPET scans) to blood tests, genetic infor-mation, neurospsychological tests, textdata from interviews and different -omics data. Additionally, data such asbackground information, as well asfinancial and feasibility constraints,need to be considered when deciding onappropriate interventions for an indi-vidual. The decision making also has asubjective component, based on per-sonal experience. This complexityeasily leads to less-than-optimal deci-sion making, even if profound clinicalexpertise is available.

In addition to the complexity and diver-sity of the data, there are issues relatedto quality and availability. Not all datais available from all patients (differentclinics have different resources, equip-ment and protocols), data quality maybe less than optimal and the data maybe in different formats and have dif-ferent properties related to equipmentand measurement environments.Clinical decision support systems(CDSS) based on principles of data-driven medicine potentially make theprocess more quantitative and objec-tive. They thus may help to providemore confidence in the decisionmaking for complex diseases.

In the EU-funded project VPH-DARE@IT [L1] a patient care platformwas developed, and subsequently vali-dated in the project PredictND [L2], thatimplements a CDSS integrating bio-markers from medical images, neu-ropsychological tests and other meas-urements. VTT and Combinostics Ltd.took care of the technical development,and clinical partners provided the needsand data for development and valida-tion. The system helps form a multi-variate integrated and easily understand-able view of a patient’s status based onmachine learning and data visualisationmethods. It uses an approach wherelarge multi-centre databases of previ-ously diagnosed patients are used tobuild mathematical models of severaldementing diseases, such as Alzheimer’sdisease, frontotemporal dementia, vas-cular dementia and dementia with Lewybodies. When a new patient arrives at aclinic, measurements are done, that arethen compared with the disease models.The software architecture enables accessto heterogeneous patient data from alarge variety of data sources in differenthospital settings.

The system’s analysis functionality hastwo main components: automatic seg-mentation and quantification of brainimages and a supervised machine

A Holistic Clinical Decision Support System

for Diagnosis of Dementia

by Mark van Gils (VTT)

Decision making for complex diseases requires efficient consideration of information from a large

variety of very different data sources. We developed a machine-learning based decision support

system for dementia diagnosis and evaluated it within four hospitals in Europe.


learning method for assessing and visu-alising the multi-variate state of patientwith respect to the investigated diseases.First, fully automated image processingmethods derive volumetric measuresfrom brain MR images. Second, thequantified imaging data are combinedwith all other available data, includingdemographic information, neu-ropsy-chological test results, blood testanalysis, CSF (cerebrospinal fluid)analysis and the patient’s genetic profile.A machine-learning paradigm, theDisease State Fingerprint [1], takes allpatient data and compares it to previ-ously diagnosed patients, providing anindex of similarity with each diseaseprofile. Moreover, it provides an interac-tive visual representation of the multi-variate patient state, allowing the clini-cian to understand which variables con-tribute to the decision suggestion, andhow each contributes (Figure 1). It canshow the overall “disease probability”,but also allows the user to zoom in to thedistributions at detailed feature level toshow, for instance, how the currentpatient’s measurements compare to thedifferent disease-related distributions inthe database. The methods work withinterval, ordinal as well as nominal data,

and have been designed from the groundup to handle issues that are important inclinical decision making. These includedealing with missing data (“incompleteinput vectors”) and the demand forexplainabality of classification results(“non black-box functioning”).

The methods have been shown, usingcross-validation, to reach a classifica-tion accuracy of 82 % when discrimi-nating patients between five differentmemory problems [2]. The clinicaldecision support tool using thesemethods was validated with 800prospective patients, to examine how itperforms, both quantitatively and froma usability perspective, with real usersat four memory clinics across Europe(Finland, Denmark, The Netherlandsand Italy) [3]. The results of this studyshowed that addition of the CDSS to theexisting clinical process affected thediagnosis and increased clinicians’ con-fidence in the diagnosis indicating thatCDSSs could aid clinicians in the differ-ential diagnosis of dementia.

This work has been co-funded by the ECunder Grant Agreements 601055 (VPH-DARE@IT) and 611005 (PredictND).

Links:

[L1] www.vph-dare.eu[L2] www.predictnd.eu

References:

[1] J. Mattila, J. Koikkalainen et al.:“Disease State Fingerprint forEvaluating the State of Alzheimer’sDisease in Patients”, J AlzheimersDis, vol 27, pp. 163-176, 2011.

[2] A. Tolonen, H. Rhodius-Meester etal.: “Data-Driven Differential Diag-nosis of Dementia Using MulticlassDisease State Index Classifier”,Frontiers in Aging Neuroscience,vol. 10, , pp. 111, 2018.

[3] M. Bruun, Marie; K.S. Frederiksenet al.: “Impact of a Clinical Deci-sion Support Tool on DementiaDiagnostics in Memory Clinics:The PredictND Validation Study”,Current Alzheimer Research, vol.16, pp. 91-101, 2019.

Please contact:

Mark van GilsVTT Technical Research Centre ofFinland Ltd., Finlandtel. +358 20 722 [email protected]

Figure�1:�A�screenshot�of�the�decision�support�tool.�On�the�left-hand�side�the�different�information�sources�are�shown,�including,�in�this�example,

MRI�imaging�data,�neuropsychological�test�results,�background�information�and�cerebrospinal�fluid�(CSF)�data.�The�middle�panel�shows�the

image�analysis�view�which�allows�for�automatic�segmentation,�quantification�and�visualisation�of�relevant�parts�of�the�brain.�The�right�hand�panel

shows�the�multi-variate�decision�support�functionality.�At�the�top,�the�various�forms�of�dementia�that�can�be�considered�are�shown,�below�is�a

“disease�state�fingerprint”�indicating�visually�and�quantitatively�to�which�disease�group�this�patient�most�likely�belongs,�together�with�the�relative

importance�of�the�different�variables.�Finally,�the�raw�data�distributions�of�separate�variables�can�be�examined�in�the�bottom�right�panel.

Appropriateness in medicine is theproper or correct use of health services,products and resources [1]. To evaluateand support appropriateness, ICT playsa crucial role, offering a growingecosystem of medical diagnostic andhealth monitoring devices, communica-tion networks, health information sys-tems, and medical data analytics. In thisscenario, data and service integration iskey to overcoming issues resulting fromheterogeneous insular systems (i.e.,information silos) and data with highcomplexity in terms of volume, variety,variability, velocity, and veracity.

The RACE project [L1] fits into thiscontext with the challenging goal ofdesigning and developing ahardware/software architecture foreffective implementation of more per-sonalised, preventive, participatory, andpredictive models of continuity of care(i.e., P4-medicine) from hospital tohome. A prototype of the overall archi-tecture has been tested over a concreteoperative scenario, demonstrating itsapplicability in the remote monitoring ofpatients with chronic moderate heartfailure (NYHA class II-III). We illustratehere some core concepts of the proposed

architecture, focusing on the specificcontribution of the University ofFlorence in the design and engineeringof Empedocle [2], an Electronic HealthRecord (EHR) system characterised byadaptability and changeability as pri-mary requirements.

The RACE architecture for continuityof careRACE is an architecture-driven projectfor remote patient monitoring whosecomponents can be organised in threemain layers, as shown in Figure 1.

Feeder layer – This level is charac-terised by HW/SW systems used byhealthcare professionals within clinicalsettings, or by non-professionals in non-clinical environments, for tracking thestate of a patient across time. On the onehand, EHR systems serve as a keyinstrument for recording, retrieving andmanipulating repositories of healthinformation in computer-processableform within clinical environments. Onthe other hand, remote monitoringdevices comprise a primary source ofinformation for health status monitoringof patients in non-clinical settings (e.g.,home), particularly in the management

of chronic diseases. They typicallyrequire sensors to measure specificphysiological parameters (e.g., bloodpressure, heart rate, pulse oximetry) andwirelessly communicate to a gatewayconnected to the Internet, so as to feedthe architecture with acquired data.Mobile health applications running onportable devices can integrate rawsensor data with higher informationprovided by non-professionals in orderto support patient self-management byimproving treatment adherence andoffering automated medicationreminders and alerts on out-of-rangemeasurements.

Integration layer – Continuity of caregives emphasis to the semantic interop-erability between multiple sources ofinformation deployed on different set-tings. To ensure a real integration ofdata (produced by the feeder layer) andservices (exposed by the healthcare ana-lytics layer), the proposed architectureexploits a middleware integration plat-form for implementing loosely-coupledpublish-subscribe communicationsbetween independently deployed andheterogeneous systems over a bus-likeinfrastructure. All moved clinical events



Connecting People, Services, and Data

for Continuity of Care

by Fulvio Patara and Enrico Vicario (University of Florence)

The RACE project (Research on Evidence-based Appropriateness in Cardiology) exploits innovative

infrastructures and integrated software services with the aim of “providing the right care, to the right

subject, at the right time, by the right provider, in the right health facility”.

Feeder layer

Healthcare analytics layer

Clinical environment

Integration layer

Clinical Decision Support System

Economic Decision Support System

…

Pathways compliance evaluation

system

Therapeutic compliance evaluation

system

Life style monitoring

system

Remote monitoring

alert system

Big data repositoryMiddleware integration platform

…

Non-clinical environment

EHR system

Remote monitoring system

… Remotemonitoring device

Health professionals

Non professionals

Mobile healthapplications

cathlaeH

reyalcs tiylanaerca

r

teomeRg nirtoinommstesyt rela

elystefiLgnirtoinom

mstesy

nosicieDlcainilCmsteySt rppouS

osicieDc imoncoEmsteySt rppouS

nom

s yawthaPcenaiplmconotiaulave

mstesy

c tiupearehTcenaiplmconotiaulave

mstesy

…c

eraweddliM

otiagrtenI

rotfaplnotiagrteni

reyalno

mr

tadag iByrtosipoer

nelcainicl-nNo

teomeR

Feeder

tnemnorivn

r layer

mnorivnelcainilC

tnem

nNoslanossiefopr

teomeRstesyg nirtoinom

… teomeRvdeg nirtoinom

mste

ceiv

ehelbioMoticaippla

thlaesno

…mstesyRHE

thlaeHslanossiefopr Figure�1:�Overview�of�the�RACE

architecture.�

are stored in a centralised big datarepository that contains the full medicalhistory about patients.

Healthcare analytics layer – A varietyof data analytics systems are built ontop of this architecture with the aim ofprocessing historical records or newinformation stored in the big data repos-itory for real-time and ex-post analyticsuses. They are organised in three maincategories based on their specific goals.Monitoring systems examine sensoreddata and generate personalisedreminders for patients, and alarms forhealthcare professionals in the case ofadverse measurements. DecisionSupport Systems (DSS) assist clinicaldecision-making tasks (i.e., clinicalDSS), as well as giving economic indi-cators to compare the costs and healthoutcomes of alternative care pathways(i.e., economic DSS). Finally, compli-ance evaluation systems evaluate theappropriateness of therapeutic treat-ment choices and care pathways.

Empedocle in action: an adaptableEHR system for continuity of careThe integration of multiple sources ofstructured information and the involve-ment of a variety of actors with differentexpertise emphasise the responsibility ofEHR systems, which become key com-ponents in driving the patient to specificcare pathways and, subsequently, inremotely monitoring the evolution of thepatient’s health status. In this context,we have developed the Empedocle EHR

system [2], a J2EE web-application thatexploits a two-level meta-modellingarchitecture based on the Reflectionarchitectural pattern [3] to combine theexpected commodity level of any EHRsystem with some specific requirementsposed by a real operative scenario ofcontinuity of care, as: agile adaptabilityof the EHR data structure to differentorganisational contexts; interoperabilityof data and services across the platform;usability by users with different spe-cialty expertise.

In such a scenario, Empedocle becomesa powerful real-time monitoring dash-board, offering to health professionalsan effective alternative to in-clinicfollow-ups, achieved by the integrationof remote monitoring data in the localadaptable EHR. Moreover, the serviceorchestration capabilities offered by theintegration platform enable severalexisting services to work together forenriching the EHR with higher-levelknowledge (e.g., diagnostic investiga-tions, drug interactions, contraindica-tions, etc. as recommended by clinicalguidelines implemented by the clinicalDSS). Given the variety of skillsinvolved in the process, connecting datawith guidelines represents a key aspectfor improving patient safety, reducingclinical risk, and evaluating the appro-priateness of care.

The RACE Consortium RACE was co-funded by TuscanyRegion (Italy) in the POR FESR 2014-

2020 program from June 2015 toSeptember 2018, and composed byindustrial partners (i.e., GPI Group,Codices, Kell, Medilogy, Spinekey, TDNuove Tecnologie), public health insti-tutes (i.e., G.Monasterio Foundation,Institute of Clinical Physiology of Pisa,Careggi University Hospital) and uni-versities (i.e., University of Florence,Sant’Anna School of AdvancedStudies).

Links:

[L1]: https://stlab.dinfo.unifi.it/race-project

References:

[1] Canadian Medical Association.“Appropriateness in Health Care”,2015.

[2] F. Patara, and E. Vicario: “Anadaptable patient-centric electronichealth record system for personal-ized home care”, in Proc. ISMICT,2014.

[3] F.Buschmann et al.: “Pattern-ori-ented Software Architecture: ASystem of Patterns”, 1996.

Please contact:

Fulvio PataraUniversity of Florence, [email protected]

Enrico VicarioUniversity of Florence, [email protected]


The use of electronic health records(EHRs) as a source of “big data” in car-diovascular research is attractinginterest and investments. IntegratingEHRs from multiple sources can poten-tially provide huge data sets for analysis.Another potentially very effective

approach is to focus more on dataquality instead of quantity. We evalu-ated the applicability of large-scale dataintegration from multiple electronicsources to produce extensive and highquality cardiovascular (CVD) pheno-type data for survival analysis and the

possible benefit of using novel machinelearning [1]. For this purpose, we inte-grated clinical data recorded by treatingphysicians with other EHR data of allconsecutive acute coronary syndrome(ACS) patients diagnosed invasively by

High Quality Phenotypic Data and Machine

Learning beat a Generic Risk Score in the

Prediction of Mortality in Acute Coronary Syndrome

by Kari Antila (VTT), Niku Oksala (Tampere University Hospital) and Jussi A. Hernesniemi (Tampere University)

We set out to find out if models developed with a hospital’s own data beat a current state-of-the art risk

predictor for mortality in acute coronary syndrome. Our data of 9,066 patients was collected and integrated

from operational clinical electronic health records. Our best classifier, XGBoost, achieved a performance of

AUC 0.890 and beat the current generic gold standard, GRACE (AUC 0.822).

coronary angiography over a 10-yearperiod (2007 -2017).

To achieve this, we generated highquality phenotype data for a retrospec-tive analysis of 9,066 consecutivepatients (95% of all patients) under-going coronary angiography for theirfirst episode of ACS in a single tertiarycare centre. Our main outcome was six-month mortality. Using regressionanalysis and machine learning methodextreme gradient boosting (XGBoost)[2], multivariable risk predictionmodels were developed in a separatetraining set (patients treated in 2007-2014 and 2017, n=7151) and validatedand compared to the Global Registry ofAcute Coronary Events (GRACE) [3]score in a validation set (patients treatedin 2015-2016, n=1771) with the fullGRACE score data available.

In the entire study population, overallsix-month mortality was 7.3 % (n=660).Many of the same variables were asso-ciated highly significantly with six-month mortality in both the regressionand XGBoost analyses, indicating gooddata quality in the training set.Observing the performance of thesemethods in the validation set revealedthat xgboost had the best predictive per-formance (AUC 0.890) when comparedto logistic regression model (AUC0.871, p=0.012 for difference in AUCs)and compared to the GRACE score(AUC 0.822, p<0.00001 for differencein AUCs) (Figure 1).

These results show that clinical data asrecorded by physicians during treat-ment and conventional EHR data can becombined to produce extensive CVDphenotype data that works effectively inthe prediction of mortality after ACS.The use of a machine learning algo-rithm such as gradient boosting leads toa more accurate prediction of mortalitywhen compared to conventional regres-sion analysis. The use of CVD pheno-type data, either by conventionallogistic regression or by machinelearning, leads to significantly moreaccurate results when compared to thehighly validated GRACE score specifi-cally designed for the prediction of six-month mortality after admission forACS. In conclusion, the use of bothhigh quality phenotypic data and novelmachine learning significantlyimproves prediction of mortality inACS over the traditional GRACE score.

This study was part of the MADDEC(Mass Data in Detection and predictionof serious adverse Events inCardiovascular diseases) project sup-ported by Business Finland researchfunding (Grant no. 4197/31/2015) asapart of a collaboration between TaysHeart Hospital, University of Tampere,VTT Technical Research CentreFinland Ltd, GE Healthcare FinlandLtd, Fimlab laboratories Ltd, BittiumLtd and Politechinco di Milano.

References:

[1] J.A. Hernesniemi, S. Mahdiani,J.A.T. Tynkkynen, et al.: “ Exten-sive phenotype data and machinelearning in prediction of mortalityin acute coronary syndrome – theMADDEC study”, 2019. Annals ofMedicine.https://doi.org/10.1080/07853890.2019.1596302

[2] T. Cheng, C. Guestrin: “XGBoost:A Scalable Tree Boosting System”,KDD ’16, 2016.https://doi.org/10.1145/2939672.2939785

[3] K. Fox, J.M. Gore, K. Eagle, et al.: “Rationale and design of thegrace (global registry of acutecoronary events) project: A multi-national registry of patients hospi-talized with acute coronary syn-dromes”, Am Heart J 141:190–199,2001.https://doi.org/10.1067/mhj.2001.112404

Please contact:

Kari AntilaVTT Technical Research Centre ofFinland ltd+358 40 834 7509



Figure�1:�Comparison�of�model�performance�by

receiving�operating�characteristic�curves�for�different

risk�prediction�models�for�six�month�mortality�among

patients�undergoing�coronary�angiography�in�Tays

Heart�Hospital�for�acute�coronary�syndrome�during

years�2015�and�2016�(n�=�1722�with�n�=�122�fatalities

during�a�six-month�follow-up).


The volume of data, the variety of datatypes, the increasing wealth of knowl-edge, and the ability to track disease andco-morbidities from start to finishalready overpower the ability of humansto make informed decision about healthand healthcare [1]. Single, personalised,user-friendly electronic health recordsfor individuals are important enablers inachieving better health services andbetter patient outcomes. However, oneof the greatest challenges in the digitalera is providing people with seamlessaccess to their health data within andacross different health systems. Digitalsolutions for healthcare are still not asinteroperable as expected and the securesharing of information is limited. Eventhough the involved stakeholders haveimplemented a big number of digitalprojects in the past twenty years in theEU, most information is still in health-care provider silos, rendering digitaltransformation for citizen empowermentdifficult to realise.

In 2017, in an open consultation con-ducted by the Commission, the majorityof respondents (93 %) either agreed (29%) or strongly agreed (64 %) with thestatement that “Citizens should be able tomanage their own health data.” Morethan 80 % of respondents believed thatsharing data could improve treatment,diagnosis and prevention of diseasesacross the EU. A large majority of respon-

dents (almost 60 %) identified the het-erogeneity of electronic health records asone of the main barriers for exchange ofhealth data in Europe [L1] . There is evi-dent public demand for secure access tohealth data across the EU.

Although individuals have the right to,and desire for, access their personaldata, including health data, most cannotyet access or securely share their healthdata seamlessly across the units of theirnational healthcare system.

In an effort to guarantee the secure andfree flow of data within the EU for publicadministrations, businesses and citizens,the new European InteroperabilityFramework (EIF) was announced in2017 [L2]. The new EIF provides guid-ance to public administrations, through aset of recommendations on interoper-ability governance, streamline processesfor end-to-end digital services, cross-organisational relationships, and newlegislation in support of interoperability.The new EIF can be adapted to supportthe eHealth domain in Europe, as acommon framework for managing inter-operability in the context of the eHealthdigital services transformation atnational level.

Within national health systems, interop-erability should occur at all four levels:legal, organisational, semantic and tech-

nical. Legal interoperability ensures thatorganisations operating under differentpolicies, legal frameworks and strategiescan work together. Organisational inter-operability refers to the way in whichpublic administrations align theirresponsibilities, business processes andexpectations to achieve mutually benefi-cial goals. Semantic interoperabilityrefers to both the meaning of data andthe exact format of the information spec-ified for exchange. Technical interoper-ability covers the applications and infra-structures linking systems and services,including interface specifications [2],data presentation and secure communi-cation protocols.

In order to secure citizens’ access to andsharing of health data, the EU is movingtowards the development of specifica-tions for a European exchange format,based on open standards, taking intoconsideration the potential use of datafor research and other purposes. Therecommendation on a European EHRexchange format sets out a frameworkto achieve secure, interoperable, cross-border access to, and exchange of, elec-tronic health data in the EU [L3]. Theaim is to deliver the right data, at theright time, for citizens and healthcareproviders, and allow for the secureaccess, sharing and exchange of EHRs.The baseline includes electronic patientsummaries, prescriptions and dispensa-

Digital Health Interoperability as a tool

towards Citizen Empowerment

by Dimitrios G. Katehakis and Angelina Kouroubali (FORTH-ICS)

When interoperability policies and consistent implementations are in place, digital tools for

citizen empowerment can be developed and used to provide information that would otherwise

be unavailable to citizens.

Figure�1:�The�New�European�Interoperability�Framework�as�a�facilitator�of�digital�transformation�for�citizen�empowerment�[1].

tions, laboratory reports, medicalimages and reports, and hospital dis-charge reports, in alignment with estab-lished priorities at a European level.

By making the citizens as patients thepoint-of-care, digital health solutions,on the one hand, target clinician empow-erment with actionable insights forfaster and accurate diagnoses and prog-noses for patient-tailored treatments,follow-ups and assistance, and, on theother hand, they assist individuals to beactive players in their health manage-ment, with timely and targeted preven-tion and assistance strategies. The com-prehensive vision focuses on makingdiseases, such as cancer [3], more pre-ventable and treatment outcome morepredictable, effective and personalised.

In order to achieve this vision, withinnational health systems, a roadmap forthe development and maintenance of

national specifications needs to be inplace. National, reusable interoper-ability specifications, compatible withthe corresponding European ones,developed through open and transparentprocesses also need to be in place,together with mechanisms and tools forcompliance control, testing and certifi-cation. Prerequisites for enabling datareuse and workflow automation includewell-defined use cases, agreed termi-nology, and reliable clinical content.

Appropriate governance and legislationare important to ensure the consistentapplication of eHealth interoperabilityand that all involved parties, includinghealth organisations and solutionproviders, will comply with it.

Links:

[L1] https://kwz.me/hy6 [L2] https://ec.europa.eu/isa2/eif_en [L3] https://kwz.me/hy7

References:

[1] A. Kouroubali, D. Katehakis: “Thenew European interoperabilityframework as a facilitator of digitaltransformation for citizen empow-erment”, Journal of BiomedicalInformatics, 94, 103166, 2019.

[2] D. G. Katehakis, et al.: “Integratedcare solutions for the citizen: Per-sonal health record functional mod-els to support interoperability”,EJBI, 13(1), 41-56, 2017.

[3] A. Kouroubali, et al.: “An Integrat-ed Approach Towards DevelopingQuality Mobile Health Apps forCancer”, in Mobile Health Applica-tions for Quality Healthcare Deliv-ery, pp. 46-71, IGI Global, 2019.

Please contact:

Dimitrios G. KatehakisFORTH-ICS, Greece+30 2810 [email protected]



Recent legislation mandates that everyhospital in Vietnam must support elec-tronic medical records [2]. This is alsoencouraged by today’s Industry 4.0. Toachieve this, a digital transformation ofthe medical field is required. Thismeans electronic medical records mustbe established, in addition to theexisting information system in eachhospital. This came into effect in allhospitals under the Ministry of Healthon March 1st, 2019, and all hospitals inVietnam are required to have electronicmedical records by the end of 2030. Asa result, a huge number of electronicmedical records are being generated andwill be available in every hospital verysoon. Compiling them thus lays thefoundations for medical case-basedresearch both within medicine andrelated fields.

Although new legislation [2] requiresthat we apply standardised technologies

across hospitals [3], existing hospitalinformation systems in Vietnam arevery diverse, owing largely to differ-ences in long-term investments ininformation technology among hospi-tals. Consequently, the development ofelectronic records has been a priorityfor some hospitals but not others. Asoutlined below, this presents huge chal-lenges when it comes to using existingelectronic medical records withexternal processing tools with the aimof gathering data to be used in research.

Firstly, the content of an electronicmedical record needs to be well definedso that all the necessary details areavailable for reference in current treat-ment procedures and future processes.Traditionally, in Vietnam, like manyother countries, hospitals have relied onpaper medical records. Transferring allinformation from paper to electronicmedical records is extremely difficult

because of problems associated withunderstanding hand-writing, time pres-sure, computer skills, etc. In addition,records must be integrative so that notonly their textual content but also theirimages and time series from medicaltests are included.

To achieve this task, as part of our ini-tial phase we investigated the use ofthe database MIMIC-III to supportVNUMED. MIMIC-III is a populardatabase which is well processed andwidely used, and we are consideringboth its database schema and practicalapplications for VNUMED. Such achoice also makes VNUMED inde-pendent of any electronic medicalrecord type in any existing hospitalinformation system in Vietnam.Furthermore, practical applicationscan then be constructed onVNUMED, hopefully like those onMIMIC-III.

towards vNuMED for Healthcare Research

Activities in vietnam

by Chau Vo, (Ho Chi Minh City University of Technology, Vietnam National University), Bao Ho (Johnvon Neumann Institute, Vietnam National University) and Hung Son Nguyen, University of Warsaw

Inspired by MIMIC-III [1], VNUMED is a unified intermediate database of electronic medical records

that is being developed in Vietnam. Its purpose is to gather medical records from hospitals, which

can be used to support medical research.


Secondly, transferring electronic med-ical records in different hospital infor-mation systems into VNUMED is a bigchallenge, stemming not only fromdiverse information technologies, butalso policies and connections betweenvarious organisations in medical andnon-medical fields. This is a complexproblem that relates to human as wellas technical issues. Data integrationalways presents its own problems, butthis situation is unique in that we aredealing with highly sensitive datarelating to many patients and organisa-tions.

Thirdly, such sensitive data must bewell protected. In MIMIC-III, the rule-based de-identification method wasused. For English data similar to thosein MIMIC-III, other more effectivelywell-defined de-identificationmethods might offer potential. In ourcase, both English and Vietnamesedata exist in VNUMED, thus, de-iden-tification on VNUMED needs to bedeveloped from scratch. Without aneffective data protection scheme,VNUMED cannot be formed – andeven if realised, VNUMED cannot beavailable for external research com-munities.

Last, but not least, once VNUMED getsstarted, post processing issues onVNUMED need to be taken into accountfor maintenance and general use. Thefirst relates mainly to the internal devel-opment of VNUMED while the latter toexternal human users and applicationprograms potential for VNUMED.Moreover, user-related policies need tobe obtained for the latter.

Development of VNUMED is expectedto be done step by step and every diffi-cult aspect will be tackled as it arises.As soon as VNUMED is available, itwill benefit researchers in a range ofmedical and non-medical fields. Duringthe development of VNUMED, we aretaking into consideration a range of pos-sible uses, including electronic medicalrecord visualisation, clinical textanalysis, drug utilisation, disease diag-nosis, etc. We anticipate that this workwill contribute to the health and well-being of the Vietnamese people, and theinternational community.

In short, an intermediate database ofelectronic Vietnamese medical records,VNUMED, is being developed to pro-vide valuable data for medical research.Many challenges lie ahead of

VNUMED and we would appreciateany input and different perspectives thatmight help us achieve our goals.

This database is being built under afive-year research project funded byVietnam National University at Ho ChiMinh City and the FIRST project ofMinistry of Science and Technology,Vietnam.

References:

[1] A. E.W. Johnson, et al.: “MIMIC-III, a freely accessible critical caredatabase”, Sci. Data 3:160035,2016. doi: 10.1038/sdata.2016.35

[2] Ministry of Health, Vietnam,Circular No. 46/2018/TT-BYT:“Regulations on ElectronicMedical Records”, December 28,2018.

[3] T. Benson and G. Grieve:“Principles of HealthInteroperability: SNOMED CT,HL7 and FHIR” (3rd Edition),Springer, 2016. ISBN 978-3-319-30370-3 (eBook).

Please contact:

Hung Son NguyenUniversity of Warsaw [email protected]

In order to tackle data integration andharmonization challenges while pre-serving privacy, we adopted anapproach based on an open, scalabledata platform for cohorts, researchersand networks. It incorporates the FAIRprinciples (Findable, Accessible,Interoperable, Reusable) [1] for optimalreuse of existing data, and builds onmaturing federated technologies[L1][L3], where sensitive data is keptlocally with only aggregate resultsbeing shared and integrated [3], in line

with key ELSI (Ethical, Legal andSocietal Issues) and governance guide-lines.

Since the measurement and observationmethods used by cohorts to collectexposures and outcomes are oftenhighly heterogeneous, using these datain a combined analysis requires thatdata descriptions are mapped onto sub-sets of research-ready core variables,and it must be clear if measurementsare similar enough to be integratively

analysed. The implemented platformnot only facilitates the process of dulycurating cohort data, but also helps pre-serve knowledge about the originalmethods used in the scope of each datacollection event, thus providing valu-able insight and a systematic frame-work to decide if and how data can bemade interoperable [2]. Althoughexpert knowledge is key to drive theharmonisation process, to some extentdata harmonisation procedures are alsosupported in the scope of the platform.

Empowering Distributed Analysis Across federated

Cohort Data Repositories Adhering to fAIR Principles

by Artur Rocha, José Pedro Ornelas, João Correia Lopes, and Rui Camacho (INESC TEC)

Novel data collection tools, methods and new techniques in biotechnology can facilitate improved health

strategies that are customised to each individual. One key challenge to achieve this is to take advantage of the

massive volumes of personal anonymous data, relating each profile to health and disease, while accounting for

high diversity in individuals, populations and environments. These data must be analysed in unison to achieve

statistical power, but presently cohort data repositories are scattered, hard to search and integrate, and data

protection and governance rules discourage central pooling.

Physically, the deployed platform mapsdown to a network of distributed datanodes, each of them in full control overtheir local users and study data, whilethe software allows data managers tocurate, describe and publish metadataabout selected datasets. New datasetscan also be derived according to agreedharmonisation dictionaries.

From a high-level perspective, datanodes can be organised in large-scale,dynamically-configured networks, withthe potential to be used in differentsetups. In a federated setting, the net-work can take form by simply intercon-necting data nodes that collaboratetowards a common goal, such as takingpart in a network of cohorts or large har-monisation studies. Since each of thenodes includes all the functionality tooperate on its own, including a publiccatalogue, one of the nodes can alsoundertake the role of gateway to othernodes, allowing more centralised gover-nance policies to be implemented (e.g. acommon catalogue entry point).

Each data node is composed of four sep-arate software components actingtogether, whose purpose is as follows:• The Data Repository is the central

data storage component at each datanode. All data operations take placehere;

• The Study Manager is where thestudies’ metadata is structured, char-acterised and eventually published;

• A Catalogue provides browsing andquerying capabilities over the pub-lished metadata;

• An Authentication Server that cen-tralises the authentication process foreach data node and provides an inter-face to manage users, groups andtheir interrelationships, as well as arole-based access control to theremaining components of the system.

One of the projects implementing thisapproach is “RECAP Preterm –Research on European Children andAdults Born Preterm” [L4], a projecthaving received funding from theEuropean Union’s Horizon 2020research and innovation programme(grant agreement No 733280) under the“networking and optimising the use ofpopulation and patient cohorts at EUlevel” topic of the “personalised medi-cine” call. RECAP Preterm includes 20partners [L5] and is coordinated byTNO (NL, ERCIM member), having

INESC TEC as leader of the workpackage responsible for implementingthe data infrastructure. The project’soverall goal is to improve the health,development and quality of life of chil-dren and adults born very preterm(VPT) or with a very low birth weight(VLBW). In order to achieve this goal,data from European cohort studies andaround the world will be combined,allowing researchers to evaluatechanges in outcomes over time whileproviding important information onhow the evolution in care and survivalof such high risk babies has changedtheir developmental outcomes andquality of life. Figure 1 presents a high-level view of RECAP Preterm networkof data nodes [L6] that is being used tostudy developmental outcomes as wellas more effective, evidence-based, per-sonalised interventions and prevention.

Also using a similar approach, therecently started EUCAN-Connect [L7]is a project having received fundingfrom the European Union’s Horizon2020 research and innovation pro-gramme (Grant Agreement No 824989)under the topic: “International flagshipcollaboration with Canada for humandata storage, integration and sharing toenable personalised medicineapproaches”. EUCAN-Connect, led byUMCG (NL) has 13 partners [L8] andaims to promote collaborative and mul-tidisciplinary research in high-valuecohort and molecular data on a largescale in order to improve statisticalpower with the aims of making new dis-coveries about the factors that impacthuman life course and facilitating theirtranslation into personalised diagnos-tics, treatment and prevention policies.

The outcome of this work will be aFAIR-compliant, federated network ofdata nodes to make cohort data findable,accessible, interoperable and reusableand enable large-scale pooled analyseswith privacy-protecting features [L3]that account for ethical, legal and soci-etal implications.

Links:

[L1] https://www.obiba.org/[L2] https://kwz.me/hyA[L3] http://www.datashield.ac.uk/[L4] https://recap-preterm.eu/[L5] https://kwz.me/hyD[L6] https://recap-preterm.inesctec.pt[L7] https://www.eucanconnect.eu/[L8] https://kwz.me/hyF

References:

[1] M.D. Wilkinson, et al.: “The FAIRGuiding Principles for scientificdata management and stewardship”,Scientific Data 3 (2016).

[2] I. Fortier, et al.: “MaelstromResearch guidelines for rigorousretrospective data harmonization”,International Journal of Epidemiol-ogy 46.1 (2017): 103-105.

[3] A. Gaye, et al.: “DataSHIELD: tak-ing the analysis to the data, not thedata to the analysis”, InternationalJournal of Epidemiology 43.6(2014): 1929-1944.

Please contact:

Artur Rocha, INESC TEC, [email protected]



Figure�1:�High-level�view�of�the�infrastructure�used�in�RECAP�Preterm.


The ageing population presents a hugechallenge for governments worldwide,which are looking for strategies to effec-tively increase the participation of olderworkers in the labour force and reducethe rates of early retirement and labourmarket exit (e.g. retirement age wasrecently raised to 67 in many EU coun-tries). Despite these efforts, in Europeearly retirement rates remain high, withthe EU-28 employment rate of 55-64year olds recorded at only 55.3 % in2016. The prevalence of chronic healthconditions in people aged 50+ is veryhigh, with every second person havinghypertension and/or another chronic dis-ease, and multimorbidity rates of 65 %

for people aged 65+ [1]. The majorityof aging workers who do choose toremain in the workforce, however, indi-cate that they plan to work past their tra-ditional retirement age, due to thereduced value of their retirement portfo-lios/income.

“Work ability” has been developed asan important multi-factorial conceptthat can be used to identify workers atrisk of an imbalance between health,personal resources and work demands[2]. An individual’s work ability isdetermined by his or her perception ofthe demands at work and their ability tocope with them. The current challenge

in using the concept is to establish ade-quate tools to evaluate and measurework ability continuously, in order tocapture the changing and evolving func-tional and cognitive capacities of theworker in various contexts.

The SmartWork project [L1], whichstarted in 2019 and will finish in 2021,aims at building a worker-centric AIsystem for work ability sustainabilityfor office workers, which integratesunobtrusive sensing and modelling ofthe worker’s state with a suite of novelservices for context and worker-awareadaptive work support. The monitoringof health, behaviour, cognitive and

SmartWork: Supporting Active and Healthy Ageing

at Work for office Workers

by Otilia Kocsis, Nikos Fakotakis and Konstantinos Moustakas (University of Patras)

SmartWork is a European project addressing a key challenge facing today’s older generation, as they are

living and working longer than their predecessors: the design and realisation of age-friendly living and

working spaces. SmartWork is building a worker-centric AI system to support active and healthy ageing at

work for older office workers. In SmartWork modelling of work ability, defined as the ability of an

individual to balance work with other aspects of their life, will account for both the resources of the

individual and factors related to work and the environment outside of work.

Figure�1:�Generic�architecture�of�the�SmartWork�suite�of�novel�services.

emotional status of the worker enablesthe functional and cognitive decline riskassessment. The holistic approach forwork ability modelling captures the atti-tudes and abilities of the ageing workerand enables decision support for per-sonalised interventions for mainte-nance/improvement of work ability.The evolving work requirements aretranslated into required abilities andcapabilities, and the adaptive workenvironment supports the older officeworker with optimised services for on-the-fly work flexibility coordination,seamless transfer of the work environ-ment between different devices and dif-ferent environments (e.g. home, office),and on-demand personalised training.The SmartWork services and module(Figure 1) also empower the employerwith decision support tools for efficienttask completion and work team optimi-sation. Formal or informal carers areenabled to continuously monitor theoverall health status, behavioural atti-tudes and risks for the people they carefor, and adapt health and lifestyle inter-ventions to the evolving worker ’sstatus.

University of Patras (Greece) is joiningforces with the Roessingh Research andDevelopment (the Netherlands) andLinköping University (Sweden) toimplement the modelling of workability in SmartWork, which will

account for both the resources of theindividual and factors related to workand the environment outside of work.The modelling of work ability will con-sider:• generic user models (groups of

users),• personalised patient models,• personalised emotion and stress

models of the office worker, • personalised cognitive models, • contextual work tasks modelling,• work motivation and values.

Continuous assessment of the variousdimensions of work ability is facilitatedthrough the continuous unobtrusivemonitoring of the health, behaviour andemotional status of the office worker(Figure 2). AI tools for prediction andrisk assessment will allow for dimen-sion specific decision support and inter-vention, such as on-the-fly flexiblework management, coping with stress atwork, on-demand training, includingmemory training.

Spark Works ITC Ltd (UnitedKingdom) will join efforts withInstituto Pedro Nunes (Portugal) for theimplementation of the UnobtrusiveSensing Framework, while Byte SA(Greece) together with Raising theFloor International (Switzerland) willimplement the Ubiquitous WorkEnvironment and Work Flexibility

tools. The European Connected HealthAlliance (Ireland) is facilitating multi-stakeholder connections around theSmartWork system. In the final sixmonths, the SmartWork system will beevaluated at two pilot sites, namely atCáritas Diocesana de Coimbra(Portugal) and at the Center for AssistedLiving Technology Heath and Care,Aarhus Municipality (Denmark). Thisis the final step towards large-scale pilotvalidation and preparation for theSmartWork system to enter the market,potentially benefiting a large number ofoffice workers, employers and formaland informal health carers.

Link:

[L1] www.smartworkproject.eu

References:

[1] M. Dyakova, A. Clarke, and H.Fraser: “Innovating care for peoplewith multiple chronic conditions inEurope project evaluation”,European Journal of Public Health26, 1, 2016.

[2] K. Tuomi, et al.: “ Promotion ofwork ability, the quality of workand retirement”, Occup Med 51,2001.

Please contact:

Otilia KocsisUniversity of Patras, [email protected]



Figure�2:�Conceptual�architecture�of�the�multi-dimensional�modelling�framework.


Populations are growing, and theaverage human lifespan is increasing.Poor lifestyle choices may develop overa long period, resulting in chronic dis-ease such as cardiovascular disease ordiabetes later in life [1]. At the sametime, ever-sophisticated wearableactivity trackers and mobile applicationsenable the assessment of the individual’sdaily habits and risk factors whichimpact their long-term health.

The WellCo European H2020 project(2017-2021), delivers a radical newinformation and communication tech-

nologies (ICT) based solution in the pro-vision of personalised advice, guidance,and follow-up for its users. Its goal is toencourage people to adopt healthierhabits that help them maintain orimprove their physical, cognitive,mental, and social well-being for as longas possible. Advice is given throughbehaviour change interventions tailoredexplicitly to each user. These interven-tions range from setting social goals torecommending activities around theseven areas defined in WellCo: cogni-tive stimulation, leisure and entertain-ment, supporting groups, physicalactivity, health status, nutrition, and tips(Figure 1). The behaviour change con-cept leverages the Behaviour ChangeWheel model [2].

WellCo provides recommendations andgoals after assessing the user’s proba-

bility of experiencing particular dis-eases. This assessment takes intoaccount the user’s profile, context,socio-economic status, health, andmental state. These characteristics arederived from data obtained from theuser’s activities of daily life (ADL) inwhich wearable sensors are seamlesslyintegrated (Figure 2), as well as fromthe user’s mood, leveraging affectivecomputing via visual and speech emo-tion recognition.

The virtual coach developed in WellCoprovides guidance and follow-up. It is

an affective-aware and always activeservice. The coach interacts throughspeech with the user to: 1) act as the vir-tual interface between the user and theplatform (managing the flow of all user-platform and platform-user interactions,and: 2) empower users in the behaviourchange process (through stimulationactivities tailored to their currentmood). A multidisciplinary team ofexperts, as well as the user’s caregivers,continuously support the service. Theyprovide their clinical evidence (expert-related outcomes) and personal knowl-edge about the user and the user ’sbehaviours (observer-related outcomes)[3] to ensure the effectiveness and accu-racy of the interventions.

The main technology-driven innovationof the WellCo solution is the health riskawareness tool. It assesses the risks of

an individual with particular precondi-tions (e.g. family history) or behav-ioural patterns (e.g. smoking) leading tothe development of chronic diseases,such as cardiovascular disease or dia-betes. The risk awareness tool also usesbehaviour and risk patterns extractedfrom the individual’s electronic healthrecords (EHR). The theoretical founda-tions for the risk awareness tool arederived from the state of the art in pre-ventive medicine. These are meta-analysis-driven, focusing on evidence-based disease risk models, includingrecent epidemiological findings. The

output of the risk awareness toolincludes: 1) relative risk, 2) modifiablerisk (the risk that can be altered bybehaviour), and 3) absolute risk (theprobability of a given disease expres-sion). The application of the risk aware-ness tool is vital in providing the indi-vidual, via the virtual coach, differentfuture “if-else” scenarios for modifiablerisks (Figure 3). They quantify how theuser’s health risk changes in response tochanges in lifestyle.

WellCo is based on a co-design processwhere end-users play a crucial role.This means that end-users are involvedright from the project’s conception, andcontribute at every step of the proto-type’s development, starting with amock-up development, via an initialproof of concept with the users, andcontinuing with three incremental pro-

WellCo: Wellbeing and Health virtual Coach

by Vlad Manea (University of Copenhagen) and Katarzyna Wac (University of Copenhagen and University of Geneva)

WellCo is a European H2020 project that aims to design and evaluate an engaging virtual coach to

help older adults make positive behavioural choices that benefit their long-term health, wellbeing,

and quality of life in physical, psychological and social interaction domains.

Figure�1:�Prototype�of�the�WellCo�mobile�application�with�information�about�activities,�social

network,�video�conversations,�and�user�profile.

Figure�2:�WellCo�dashboard�and�two�models

of�wearable�device�compatible�with�WellCo.

totypes to ensure a valid and effectiveservice integration and validation.

The expected outcomes of WellCo maybe categorised as short-term (feelingmotivated to change one’s habits), inter-mediate (e.g. changing attitudes, normsand behaviours; translating good inten-tions into practical actions), and long-term (initiating and maintaining posi-tive habits, changing activities andimproving health status and quality oflife over the long term). To demonstratethe health behaviour change process,long-term trials have just started (mid-2019) in several European countries,including Italy, Spain, and Denmark,with the participation of public healthand social care organisations (Italy andSpain) and research organisations(Denmark).

The project involves eight Europeanpartners: HI-Iberia (Spain, the coordi-nator), Fondazione Bruno Kessler(Italy), Institut Jozef Stefan (Slovenia),Gerencia de Servicios Sociales deCastilla y León (Spain), ConnectedCareServices B.V. (The Netherlands),Monsenso (Denmark), SyddanskUniversitet (Denmark), andKøbenhavns Universitet (Denmark).

Links:

[L1] http://wellco-project.eu/[L2] http://qualityoflifetechnologies.org

References:

[1] M. Naghavi et al.: “Global, region-al, and national age-sex specificmortality for 264 causes of death,1980–2016: a systematic analysisfor the Global Burden of DiseaseStudy 2016,” Lancet, vol. 390, no.10100, pp. 1151–1210, Sep. 2017.

[2] S. Michie, M. M. van Stralen, andR. West: “The behaviour changewheel: A new method for charac-terising and designing behaviourchange interventions,” Implement.Sci., vol. 6, no. 1, p. 42, Dec. 2011.

[3] N. E. Mayo, et al.: “MontrealAccord on patient-reported out-comes use series—paper 2: termi-nology proposed to measure whatmatters in health,” J. Clin. Epi-demiol., Apr. 2017.

Please contact:

Vlad Manea, QoL Lab, University ofCopenhagen, [email protected]



Figure�3:�WellCo�risk�assessment�scenarios:�1.�“if�the�person�continues�like�this…”,�2.�if�they

quit�smoking,�and�3.�if�their�blood�pressure�decreases.


ICT technologies have great potential tohelp prolong the time elderly people canlive independently in their preferredenvironment. However, several barriersexist when it comes to seniors actuallyusing existing technologies: older adultsmay have different needs and attitudesto younger generations - for example,attitudes towards technology, physicalor cognitive limitations, and interests.Furthermore, the needs of elderlypeople are not static; they are likely toevolve over time depending on vari-ables such as evolving health conditionsand preferences. Thus, the personalisa-

tion of ICT-based support is a funda-mental challenge.

In the AAL PETAL project [L1] at theHIIS Laboratory of CNR-ISTI [L2] weare developing a platform for personal-ising remote assistance of older adultswith mild cognitive impairments, with aparticular focus on the support oflighting systems in order to provide ori-entation over time and in space. Thiscategory of users suffers from cognitiveissues, such as the tendency to forgettasks/events and/or other issues such ascardiovascular issues, reduced sight,

and irregular eating habits, often associ-ated with increased risk of social isola-tion and depression. The platform moni-tors the user’s environment and behav-iour, and personalises applications andcontrol devices to better support seniorsin their daily lives. Thus it exploitssmart objects, such as lights, to provideappropriate activation or relaxationstimuli, and it generates alerts andreminders for physical and social activi-ties, orientation over time and space,and sleep quality.

The user or caregiver can set the deviceto control lights and other digitaldevices when relevant events occur. Inthis way it is possible to personalisecontrol of the lights and other digitalappliances, to set personalised warningmessages to be issued in risky situa-tions, and persuasive messages toencourage healthier habits (e.g., morephysical activity). The possible person-alisations are expressed in terms ofsimple trigger-action rules [1, 2, 3].Triggers represent situations or eventsthat might be useful for caregivers toknow: e.g. health/cognitive/emotionalstatus, cognitive/physical/socialactivity, especially when the caregiveris not present (remote monitoring). Theinformation associated with triggers isderived from various sensors (e.g.motion, proximity, lights, noise, respira-tion, heart). Actions represent what thetechnological equipment within thehome could do: control appliances (e.g.switch on/off lights, close/open doors,play tv/radio), send reminders, sendalarms, provide information about theuser’s needs. Personalised rules that canbe obtained with this approach include:• When the user leaves the house and

rain is forecast, a phone alert can sug-gest taking the umbrella;

• A message can be sent to the caregiv-er when the user leaves home duringthe night;

• When a caregiver sends a message“where?”, an automatic answer pro-vides the user’s location;

A Personalisation Platform for older Adults

with Mild Cognitive Impairments

by Marco Manca, Parvaneh Parvin, Fabio Paternò, Carmen Santoro and Eleonora Zedda (ISTI-CNR)

The AAL PETAL project has developed a platform for personalising remote assistance of older adults

with mild cognitive impairments. The platform is targeted at caregivers without programming

knowledge in order to help seniors in their daily activities at home.

Figure�1:�

An�example�of�home

involved�in�the�trials.�

Figure�2:�Sensors�and�Objects�in�the�Trials.�

• If a user has not done the plannedcognitive exercises, a blink of thelight acts as a reminder to continuethis activity;

• If the user is close to the living roomand time is 4 pm, turn on the TV.

We have organised trials in homes witholder adults with mild cognitive impair-ments, distributed across variousEuropean countries (Figure 1 shows onehome involved in the trials). Figure 2shows the types of sensors, objects, anddevices we use in such trials. Each userhas a tablet, and a smartwatch. We haveselected a smartwatch that, in additionto detecting physiological parameterssuch as heart rate and step counter, isable to connect and communicate at thesame time through Bluetooth and Wi-Fi. This is exploited to obtain indoorpositioning with the support of prox-imity beacons. For light, we use theGREAT luminaire (designed byBartenbach, a company involved in thePETAL project), which aims to providehealth-stimulating, biorhythm-stabil-ising, high-quality light for high visualdemands, and creates an activating orcalming room ambiance with differentlight scenes. The extremely high lightintensity (1000 lx at eye level) of the

GREAT luminaire leads to effects com-parable with classic light therapy withinfive hours of use. It compensatesmissing daylight and provides distrib-uted light within a whole room of about16 m2. In addition, we use various typesof Philips Hue lights to support similareffects in other parts of the home. Othersensors detect gas, smoke, humidity,use of objects, whether windows ordoors are open and so on. The platformis also connected with an app developedby Ideable (a Spanish companyinvolved in the PETAL project), whichsupports serious games for cognitivestimulation in order to allow caregiversto define rules depending on the user’scognitive and emotional state ordepending on the training results.

Supporting this technology is a person-alisation platform that includes a mid-dleware (context manager) to gatherraw information from the various sen-sors and convert it into data that can beanalysed in terms of logical events andconditions. In this way, when the per-sonalisation rules are created by theuser or caregiver, it is possible to detectdynamically when they should be trig-gered and the consequent actions per-formed.

Links:

[L1] http://www.aal-petal.eu/[L2] https://giove.isti.cnr.it/lab/home

References:

[1] G. Ghiani, et al.: “Personalizationof Context-dependent Applicationsthrough Trigger-Action Rules”,ACM Transactions on Computer-Human Interaction, Vol.24, Issue 2,Article N.14, April 2017.

[2] H. Huang, M. Cakmak:“Supporting mental modelaccuracy in trigger-actionprogramming”, in Proc. ofUbiComp '15. ACM, New York,NY, USA, 215-225. DOI:http://dx.doi.org/10.1145/2750858.2805830

[3] B. Ur, et al.: “Practical trigger-action programming in the smarthome”, CHI 2014: 803-812, 2014.

Please contact:

Fabio Paternò, ISTI-CNR, Italy+39 050 315 [email protected]



The progress and the use of moderntechnologies and digital services havechanged the way people monitor healthand well-being. The ability to accesspersonal data for remote monitoring andself-management can greatly improvehealthcare.

Broadly speaking, there are two aspectsto digital health: the perspective of thepatient, and that of the state. For thepatient, digital health represents person-alised and higher quality care that adaptsto their needs. For the state, digitalhealth can reduce inefficiencies inhealthcare provision and costs innational health systems. Thus, digital

health has resulted in a significantincrease in the quality of life of patients,as well as a significant positive effect onthe global economy.

The rise of digital health raises theurgent need for technology to support"smart care" in the home environment,enabling people to live independently,for longer, in their preferred environ-ment, whilst offering their physiciansand carers resources and tools tomanage their patients effectively andefficiently.

Many factors contribute to the develop-ment of neurodegenerative diseases,

which mainly affect people over the ageof 60. Some of the most common neu-rodegenerative diseases includedementia (with its most well-knownform being Alzheimer's disease),Parkinson's disease, and epilepsy.Several digital health systems havebeen proposed to support people suf-fering from neurodegenerative diseases,with an emphasis on dementia, since itis the most common neurodegenerativedisease. The total population withdementia is projected to reach 82 mil-lion in 2030 and 152 million by 2050[L1]. The increasing prevalence of thisdisease is putting pressure on nationalhealth systems worldwide.

technological Memory Aids for Neurodegenerative

Diseases and the AuDi-o-Mentia Approach

by Eleni Boumpa and Athanasios Kakarountas (University of Thessaly)

Rates of dementia are increasing, putting pressure on national health systems. Digital health can help both

patients and national health systems in a range of ways. One technology that is being developed is AuDi-o-

Mentia,an acoustic memory aid to help people in the early stages of dementia.


Just like other digital health systems,the goal of proposed digital health sys-tems in the area of dementia is to pro-vide personalised care for patients andtheir relatives and/or caregivers, tomonitor their mental health and well-being, and to give feedback on theirhealth condition to their doctors. Theproposed systems are mainly focusedon either the early diagnosis ofdementia symptoms or the provision ofservices to support the sufferers. Thus,these proposed systems can assistpeople in their daily lives, monitor theirmood, guide them in their daily routine,supervise them when they leave theirhome environment, give themreminders (for example, to take theirmedication), and help them maintainactive social lives and avoid becomingisolated because of their disease.

A promising technology for people suf-fering from dementia, and their rela-tives, is the AuDi-o-Mentia project[L2]. AuDi-o-Mentia is a home assis-tive system that works with the use ofsound stimuli. The sound stimulus wasselected because is beneficial for thesufferers and help them recall theiridentity, with better results than thoseproduced by other stimuli, like anoptical stimulus [1, 2]. Since this wasproved by professionals (i.e., neurolo-gists), exploiting music-therapy ses-sions, there was the need for an imple-mentation that would be easily inte-grated in homes. AuDi-o-Mentia’s func-tion is to reproduce a distinctive soundrepresenting each of the familiar facesof the sufferers, in order to providethem with an additional stimulus torecognise their loved ones. Wheneverone of the familiar faces enters the suf-

ferer’s home, the associated character-istic sound will be produced. The userselects the sound that represents eachfamiliar, since the user has differentmemories from each person [3].Alternatively, a caregiver may createthe appropriate acoustic stimuli associa-tions with the sufferer’s familiar people,exploiting music therapy techniques.

AuDi-o-Mentia is based on the basicrules of music therapy, aiming to stimu-late the auditory memory, which is sta-tistically one of the last parts of memorythat will be affected by a degenerativedisease. It is a universal system, inde-pendent of age, gender, nationality oreconomic status. The concept itself isbased on distributed smart speakers atthe sufferer’s home and the stimulationof memory depending of the identity ofthe visitor. The system is transparent tothe sufferer, avoiding any confusion.The interface is based on the identifica-tion of the visitor (exploiting RFID, orID detection via WiFi or Bluetooth) bythe system and the acoustic stimulation.Thus, the interaction is made physicaland no special training is required. Thismakes it suitable for people sufferingfrom degenerative diseases and simpleenough for caregivers to use.

In summary, digital health is the key toproviding personalised health servicesand confronting many challenges facingnational health systems. Smart tech-nology is expected to provide novelsolutions to sufferers and new tools tocaregivers. It is the role of researchersto creatively apply technologies andknowledge to form new solutions forour fellow humans.

Links:

[L1] https://kwz.me/hy8[L2] https://audiomentia.com/

References:

[1] McDermott, et al.: “Theimportance of music for peoplewith dementia: The perspectives ofpeople with dementia, familycarers, staff and music therapists”,Aging Ment. Health 2014, 18,706-716.

[2] Holmes, et al.: “Keep music live:Music and the alleviation of apathyin dementia subjects”, Int.Psychogeriatr. 2006, 18, 623-630.

[3] E. Boumpa, et al.:“An Acoustic-Based Smart Home System forPeople Suffering from Dementia”,Technologies 7.1 (2019): 29.

Please contact:

Eleni Boumpa, AthanasiosKakarountas, University of Thessaly, Lamia, Greece [email protected], [email protected]

Figure�1:Detection�of�a

familiar�person�and

reproduction�of�the

associated�acoustic�stimulus

at�the�smart�speaker�located

closest�to�the�sufferer.

Recent advances in information tech-nology make it easy for businesses andother organisations to collect largeamounts of data and use data analyticstechniques to derive valuable informa-tion and improve predictions. The infor-mation obtained, however, is usuallysensitive, and may endanger the privacyof data subjects. Whilst the GeneralData Protection Regulation (GDPR)necessitates a technological means toprotect privacy, it is vital that this isachieved in a way that still allowshealthcare stakeholders to extract mean-ingful information and make good pre-dictions (e.g., about diseases). ThePAPAYA project aims to provide solu-tions that minimise privacy risks whileincreasing trust in third-party dataprocessors and the utility of the under-lying analytics.

The newly developed PAPAYA platformwill integrate several privacy-pre-serving data analytics modules,ensuring compliance with the GDPR.The project considers different settingsinvolving various actors (single/mul-tiple data sources, queriers) andensuring different privacy levels. Theproject will facilitate user experiencefor data subjects while providing trans-parency and control measures.

The PAPAYA project focuses on threemain data analytics techniques, namely,neural networks (training and classifica-tion), clustering, and basic statistics(counting) and aims at developing theirprivacy - preserving variants while opti-mising the resulting performance over-head and assuring an acceptableutility/accuracy. More specifically, pri-vacy-preserving neural networks(inspired by the architecture of neuronsin human brains) learn predictionmodels about a certain characteristic/capability using some test datasets and

further apply this model over new datato make accurate predictions whilekeeping the input data confidential. Onthe other hand, privacy-preserving clus-tering algorithms allow data owners togroup similar (but confidential) dataobjects in clusters. Finally, privacy-pre-serving counting primitives enable par-ties to encrypt one or several datasetsrelated to individuals and further countthe number of individuals in the set. Themain cryptographic tools that will be

used to design these new solutions arehomomorphic encryption, secure multi-party computation, differential privacyand functional encryption.

Privacy-preserving neural networksfor two digital health use casesThe PAPAYA project defines two digitalhealth use cases, namely privacy-pre-serving arrhythmia detection and pri-vacy-preserving stress detection. Whileboth use cases rely on neural networks,the former (arrhythmia detection) onlyconsiders the classification phase andthe latter (stress detection) involvesmultiple data sources, such as hospitals,that collaboratively train a stress detec-tion neural network model. Both use

cases and the underlying PAPAYA solu-tions are summarised below.

Privacy-preserving arrhythmiadetectionThis use case targets scenarios wherebypatients need to perform cardiac param-eters analyses with the goal of verifyingthe presence/absence of arrhythmia.The patient wears a device that collectshis/her ECG data for a fixed amount oftime (e.g., 24 hours). Once the patient

returns the device to the pharmacy, theECG data are protected and submittedto the PAPAYA platform, as illustratedin Figure 1. The data are then analysedto predict whether the patient suffersfrom arrhythmia.

The project aims to develop a privacy-preserving classification solutionwhereby the neural network model isexecuted over confidential data. Thesesolutions use advanced cryptographicschemes such as homomorphic encryp-tion [1] or secure multiparty computa-tion [2]. The main challenge in usingsuch tools is the complexity of theneural network in terms of size and theunderlying operations. Therefore,



PAPAYA: A Platform for Privacy Preserving Data

Analytics

by Eleonora Ciceri (MediaClinics Italia), Marco Mosconi (MediaClinics Italia), Melek Önen (EURECOM)and Orhan Ermis (EURECOM)

The PAPAYA project is developing a dedicated platform to address privacy concerns when data

analytics tasks are performed by untrusted data processors. This platform regrouping will allow

stakeholders to ensure their clients’ privacy and comply with the General Data Protection Regulation

(GDPR) [L1] while extracting valuable and meaningful information from the analysed data. PAPAYA

targets two digital health use cases, namely arrhythmia detection and stress detection, whereby

patients’ data are protected through dedicated privacy enhancing technologies.

Figure�1:�A�patient’s�ECG�data�are�collected�by�a�pharmacist,�sent�to�a�trusted�cloud,�protected

and�submitted�to�the�PAPAYA�platform�to�predict�arrhythmia.�The�detected�arrhythmia�are�used

by�a�cardiologist�to�redact�the�patient’s�report.�

PAPAYA will use these advanced cryp-tographic tools once the original neuralnetwork is modified in order to make itcompatible with the actual crypto-graphic tool (for example, complexoperations are approximated to lowdegree polynomials). This modifiedneural network will still maintain agood level of accuracy.

Privacy-preserving stressmanagementThis use case targets workers whosuffer from stress. It would be veryhelpful to have an automatic solutionthat would help anxious and stressedpeople to recognise symptoms at theironset and suggest mitigation strategiesto help the person take preventativeaction and keep stress levels in check.To this end, sensitive health data fromIoT sensors are collected by multiple

sources and used to train a collaborativemodel via the PAPAYA platform asshown in Figure 2, with the goal ofautomatically detecting stress condi-tions in workers.

As a potential solution for this use case,we are studying the problem of privacy-preserving collaborative training basedon differential privacy [3] involvingmany data owners who need to jointlyconstruct a neural network model.Differential privacy prevents partici-pants’ individual datasets from beingleaked, but allows the joint model to becomputed.

This project is a joint work of thePAPAYA project consortium. ThePAPAYA project is funded by the H2020Framework of the European Commissionunder grant agreement no. 786767. In this


Figure�2:�Health-related�parameters�are�collected�from�workers,�aggregated�locally�and

outsourced�to�train�a�collaborative�model,�which�can�be�later�used�to�perform�real-time�detection

of�stress�and�anxiety�conditions.�

project, six renowned research institutionsand industrial players with balancedexpertise in all technical aspects of bothapplied cryptography, privacy andmachine learning are working together toaddress the challenges of the project:EURECOM (project coordinator), AtosSpain, IBM Research Israel, KarlstadUniversity Sweden, MediaClinics Italiaand Orange France.

Link:

[L1] https://kwz.me/hyK

References:

[1] R. Rivest, L. Adleman and M. L.Dertouzos: “On data banks andprivacy homomorphisms,”Foundations of secure computation,pp. 169--180, 1978.

[2] A. Chi-Chih Yao: “Protocols forsecure computations” (extendedabstract), in 23rd AnnualSymposium on Foundations ofComputer Science, Chicago,Illinois, USA, 1982, p.160–164,n.a., 1982. IEEE Computer Society.

[3] C. Dwork: “Differential privacy”,in Proc. of the 33rd InternationalConference on Automata,Languages and Programming -Volume Part II, ICALP’06, pages1–12, Springer, 2006.

Please contact:

Orhan ErmisEURECOM, [email protected]

Future mHealth informatics rely oninnovative technologies and systems fortransparent and continuous collection ofevidence-based medical information atanytime, anywhere, regardless of cov-erage and availability of communicationmeans. Such an emerging critical infra-

structure is influenced by factors suchas biomedical and clinical incentives,advances in mobile telecommunica-tions, information technology develop-ments, and the socioeconomic environ-ment. This cross dependency has led toconcerns about reliability and resilience

of current network deployments, henceit is imperative that communication net-works be designed to adequatelyrespond to failures, especially in cloud,mobile and Internet Of Things (IoT) /Web Of Things (WoT) environmentsthat have traditional boundaries.

Resilient Network Services for Critical mHealth

Applications over 5G Mobile Network technologies

by Emmanouil G. Spanakis and Vangelis Sakkalis (FORTH-ICS)

DAPHNE is aiming to develop a resilient networking service for critical related applications, as a novel

approach for next generation mHealth information exchange. Our goal is to provide in-transit persistent

information storage, allowing the uninterruptible provision of crucial services. Our system will overcome

network instabilities, capacity efficiency problems, incompatibilities, or even absence of end-to-end

homogeneous connectivity, with an emphasis on future networks and services (i.e. 5G). We aim to provide

a set of tools for the appropriate management of communication networks during their design time and

avoid the “build it first, manage later” paradigm.

New community-based arrangementsand novel technologies can empowerindividuals to be active participants intheir health maintenance by enablingthem to self-regulate and control theirwellness and make better lifestyle deci-sions using community-based resourcesand services. Mobile sensing tech-nology and health systems, responsiveto individual profiles, supported byintelligent networking infrastructuresand combined with cloud/IoT com-puting can expand innovation for newtypes of interoperable services that areconsumer oriented and communitybased. This could fuel a paradigm shiftin the way health care can be, or shouldbe, provided and received, whilereducing the burden on exhaustedhealth and social care systems [1].

Crucial innovation is needed to makeand deploy large scale ICT that facili-tates end-user services that are usable,trusted, accepted and enjoyed. This willrequire multi-domain, multilevel, trans-disciplinary work that is grounded intheory and matched by business abilityto bring innovation to the market.Importantly, it must be driven by theneeds, expectations and capabilities ofindividuals and healthcare profes-sionals. Communication networks areone of the most important critical infra-structures underpinning this system,since many other critical infrastructuresdepend on them in order to function.The heavy reliance on communicationnetworks has led to concerns about reli-ability and resilience; hence, it is imper-ative that such networks are designed to

adequately respond to failures andattacks, especially in environments thathave traditional boundaries. The goal isto form services at scale, establishing alayer of trust among entities in order toshare/collaborate/communicate whileminimising the likelihood of failure.

The fifth generation of mobile tech-nology (5G) is positioned to address thedemands and business context of 2020and beyond. It is expected to enable afully mobile and connected society andto empower socio-economic transfor-mations in countless ways, many ofwhich are unimagined today, such asfacilitating productivity, sustainabilityand well-being. The meaning of 5G, andthe ways it will affect electronic healthservices, is still a subject of discussionin the industry. However, the softwari-sation of networks is expected to shapeits design, operation and management.Right now there is a growingdensity/volume of traffic and a rapidlygrowing need for connectivity. To facil-itate this, a multi-layer densification isrequired, as well as a broad range of usecases and business models, in order forvendors to avoid the “build it first,manage it later” paradigm. In thisproject our goal is to extend the per-formance envelope of 5G networksincluding embedded flexibility, a highlevel of convergence and access in ahighly heterogeneous environment(characterised by the existence of mul-tiple types of access technologies,multi-layer networks, multiple types ofdevices, multiple types of user interac-tions, etc).

DAPHNE is implemented around abundle protocol (BP) (IETF RFC 4838and RFC 5050) adapted to 5G networkstack implemented around a conver-gence layer (CL). Data packets areencapsulated in the BP and can trans-parently travel across regions with dif-ferent network protocol stacks. Ourconvergence layer implementationsmay include HTTP, TCP, UDP,Ethernet, BT & BLE, AX.25, RS232,IEEE 802.11x, 802.15.4, LR-WPAN,5G and other. In our reference imple-mentation, we created a “dtntunnelproxy” forming a DTN tunnel over aheterogeneous 5G network that can sus-tain any delay or disruption, thanks tothe in-network storage of our designedarchitecture.

Daphne [L1], implements a resilientservice for critical to support mHealthservices enabling personalisation,patient inclusion and empowermentwith the expectation that such systemswill enhance traditional care in a crisisand provide provision in a variety ofsituations, where remote consultationand monitoring can be implementeddespite the lack of end-to-end connec-tivity (Figure 1) [2]. In this scenario weenvision next generation personalhealth systems and pervasive mobilemonitoring to empower individuals inwell-being and disease prevention, andchronic disease management. IoMTand Personal Health Systems coveringwell-being, prevention of specific dis-eases or follow-up and management ofexisting chronic diseases can enhancepatient empowerment and self-caremanagement.

DAPHNE is focusing on the underlinecyber-physical ecosystem of intercon-nected sensors and actuators to regulatethis networking ecosystem, formed bya collection of biomedical sensors,wearable medical devices, control/sinknodes (mobile phones) and gatewayssupporting underline critical healthcareservice and enable intelligent decisionmaking. These proposed technologiesfor the underlying architecture, embraceremote monitoring, sensor data collec-tion, remote patient monitoring, extrac-tion of health related features for detec-tion of risks/ alarming and/or alerting,personalized feedback and recommen-dation services for the patient orinformal caregiver. The growing devel-opments in the IoMT, including smartconnected technology, can be used for



Figure�1:�Daphne�mHealth�service�network.


smart and uninterrupted data collectionin order to benefit healthcare and itsdata-processing abilities, timely deci-sion-making and the overall goal forbetter patient outcomes (i.e. remotehealthcare and monitoring, better drugmanagement through smart devices andactuators, adjustment of therapies andtreatment plans, medical device moni-toring and control, management ofdevices within critical healthcare infra-structures).The ability to resiliently pro-vide in-transit persistent informationstorage will allow the uninterruptibleprovision of crucial e-Health services,overcoming network instabilities,incompatibilities, or even absence, for along duration. In our implementation wesupport application scenarios for:Heterogeneous networks, Harsh inter-mittent connectivity, Extremely largedelays and, Severe disruptions. Ourfocus is on the integration of a prototypeproxy implementation adapted formHealth requirements and futureinternet services through emerging

telecommunication converging net-works (i.e. 5G) [3]. We analyze the vul-nerabilities from a fault tolerant perspec-tive, while taking into account the auto-nomic principles and we propose a self-healing based framework for 5G net-works to ensure availability of servicesand resources. We will emphasise theproblem of reliable system operationwith extremely low power consumptionand discontinuous connectivity, whichare typical for continuous monitoring ofpeople. The goal is to study networkfailures making them imperceptible byproviding service continuity and by min-imising congestion.

This project has received funding fromthe Hellenic Foundation for Researchand Innovation (HFRI) and the GeneralSecretariat for Research and Technology(GSRT), under grant agreement No1337.

Link:

[L1]: https://daphne.ics.forth.gr/

References:

[1] E.G.Spanakis et al.: “Technology-Based Innovations to FosterPersonalized Healthy Lifestyles andWell-Being: A Targeted Review”, JMed Internet Res 2016;18(6):e128,DOI: 10.2196/jmir.4863, PMID:27342137.

[2] E.G. Spanakis, A.G. Voyiatzis:“DAPHNE: A Disruption-TolerantApplication Proxy for e-HealthNetwork Environments”, 3rdInternational Conference onWireless Mobile Communicationand Healthcare, Paris, France,November 21-23, 2012.

[3] E.G. Spanakis: “Internet of MedicalThings for Healthcare in SmartMonitoring and DiverseNetworking Environments”, EMBC2018.

Please contact:

Emmanouil G. Spanakis, FORTH-ICS, [email protected]

The ERCIM postdoctoral fellowship programme is open toyoung researchers from all over the world. It focuses on abroad range of fields in Computer Science and AppliedMathematics. The fellowship scheme also helps young sci-entists to improve their knowledge of European researchstructures and networks and to gain more insight into theworking conditions of leading European research institu-

tions. The fellowships are of 12 months duration (with apossible extension), spent in one of the ERCIM memberinstitutes. Fellows can apply for second year in a differentinstitute.

Why to apply for an ERCIM Fellowship?The Fellowship Programme enables bright young scientistsfrom all over the world to work on a challenging problem asfellows of leading European research centers. An ERCIMFellowship helps widen the network of personal relationsand understanding among scientists. The programme offersthe opportunity to ERCIM fellows:• to work with internationally recognized experts;• to improve their knowledge about European research

structures and networks;• to become familiarized with working conditions in lead-

ing European research centres;• to promote cross-fertilization and cooperation, through

the fellowships, between research groups working in sim-ilar areas in different laboratories.

Deadlines for applications are currently 30 April and 30 September each year.

Since its inception in 1991, over 500 fellows have passedthrough the programme. In 2005 the Fellowship Programmewas named in honour of Alain Bensoussan, former presidentof Inria, one of the three ERCIM founding institutes.

https://fellowship.ercim.eu

ERCIM “Alain bensoussan” fellowship Programme

It was a holistic experience, and therealization of the experience andknowledge I gained only grows each time Ilook back on it. ERCIM fellowship is aunique opportunity that needs to begrasped, that shapes your future and cangive you experiences that will inspire youfor many years.

Naveed Anwar BHATTIFormer ERCIM Fellow

The ERCIM Postdoc was a greatopportunity! I moved to anothercountry, I met new people and Iworked in machine learningapplied to industry. This definitelychanged my career. Thanks a lot!

Rita Duarte PIMENTELFormer ERCIM Fellow

Research and Innovation


Eu

rop

ea

n R

ese

arc

h a

nd

In

no

va

tio

n

A Contractarian Ethical

framework for Developing

Autonomous vehicles

by Mihály Héder(MTA SZTAKI)

The way forward for autonomous vehicle ethics does not

revolve around solving old moral dilemmas, but on

agreeing on new rules.

Contractarian ethical frameworks claim that the norms weaccept as good or proper are mere results of social compro-mise that is ultimately driven by the self-interest of theinvolved parties. This position is in contrast with other para-digms around the foundations of ethics, for instance virtuesor divine commands.

We are under no obligation to subscribe to one single, exclu-sive ethical paradigm for all purposes and aspects of ourlives. One could apply a particular approach to autonomouscars while allowing others in other domains as long as theycan be made compatible.

We believe that a contractarian approach should be taken inthe context of autonomous cars, and also that if we are toever enjoy a serious diffusion of fully autonomous cars it willhappen based on the grounds of compromise - or it won’thappen at all.

From this it follows that the decisions required duringautonomous car development are to be found at the intersec-tion of what is generally considered to constitute acceptablevehicle behaviour as applies to all road users - if such an inter-section exists at all. This means that the industry involved indefining such behaviour should simply make proposals andask for a compromise rather than chasing for moral truths.

The case of autonomous cars should be easier than othersocial issues, too, because any person can conceivably takeon the identity of any type of road user in a particular situa-tion. An individual may be a pedestrian in the morning, abicycle rider during the day and a passenger in the eveninge.g. in an autonomous cab. With other issues our identitiestend to be more entrenched.

Let us take most basic autonomous vehicle related ethicaldilemma to illustrate the approach. The autonomous car findsitself in an emergency situation in which it can either hit andkill a group of pedestrians or swerve and sacrifice its passen-gers [1]. There appears to be no other option. This thoughtexperiment has been advanced with a variety of discrimi-nating factors like the number of casualties inpedestrians/passengers, age, gender, various forms of socialrole of the involved people, etc.

The example reveals the very high dependence on both ourand the car’s epistemic facilities in evaluating such situations.

In reality the car cannot be certain what kind of objects it hasdetected as the Arizona Uber incident in which a cyclist diedillustrated. Worse still, it has only a partial appraisal of the


uncertainty of the object categorisation itself. Also, it hasbeen shown that the neural networks - the technology thatperforms the identification - can be tricked [2] (the resilienceof neural networks to such attacks is a research subject at ourdepartment). The uncertainty attached to such situationsmeans that the ethical dilemma itself is only known proba-bilistically.

At any rate, we are not expecting moral agency from the caritself. Instead these decisions are supposed to be madedesign-time. Here is where the fallacy of our epistemic facil-ities come into play. When asked in an experiment a largemajority of subjects will say that the vehicle should sacrificeone to save many. But such preventive action has the non-trivial consequence that this known vehicle behaviour allowsfor malicious actors to trick cars into killing people - by actu-ally jumping in front of a car or even without if the objectdetection can be tricked. Or, the pedestrian might jump awaybut the vehicle happens to swerve in the same direction,causing the very tragedy it tried to prevent. When presentingsuch scenarios to subjects they often backtrack on their pre-vious opinions. Nontrivial consequences are one reason whysurveys like the Moral Machine [3] are flawed.

Let us instead entertain a typically contractarian proposal:the autonomous car shall brake intensely in such situationsbut it will never swerve. This proposal has the marks of goodrule-based systems: it is both simple to implement and tounderstand and results in predictable behaviour.

Such a proposal, as long as we think in the context of the cur-rent traffic conditions, would result in tragic casualties insome individual cases, which might have arguably been pre-vented by a human driver. However, the simplicity of such aself-preserving rule will allow those very conditions to bechanged so that the situation won’t arise.

The contractarian approach is rational because it does notattempt to solve moral value dilemmas that have proven to beintractable over the last couple of hundred years. It alsoaccounts for the unimaginability of future situations that isthe reality of design-time work. What it does instead is come

up with a simple set of rules design-time, asking for the con-sent of all road users, and thereby in run-time it allows formore control of the situations that impact humans by virtueof being easily predictable. This also allows an evolution ofthe overall attitude of human road users towards autonomousvehicles in yet unforeseen ways to manage their presence intheir own self-interest.

Finally, in order for the contractarian approach to work itneeds to stick to its principles - beyond simplicity and intelli-gibility, those behaviour patterns should be well-known oreven advertised; it should be accepted if not with full con-sensus but at least with compromise; and these behaviourpatterns should be guaranteed to operate consistently asmuch as possible. About a hundred years ago, when the auto-mobile was a novelty, pedestrians needed to vacate someparts of the streets in ways they were not required to in theage of horse carriages - but in return they got traffic lights. Ata red light, drivers stop even if there is absolutely no trafficfor kilometres: the contract is binding and ensures safety bynot allowing any self-judged overruling.

This work was supported by the Bolyai scholarship of theHungarian Academy of Sciences and by the ÚNKP-18-4New National Excellence Program of the Ministry of HumanCapacities.

References:

[1] P. Lin: “Why Ethics Matters for Autonomous Cars”, M.Maurer et al. (Hrsg.), Autonomes Fahren, DOI10.1007/978-3-662-45854-9_4, 2015.

[2] A. Nguyen, J. Yosinski, J. Clune: “Deep Neural Net-works are Easily Fooled: High Confidence Predictionsfor Unrecognizable Images”, in Computer Vision andPattern Recognition (CVPR '15), IEEE, 2015.

[3] E. Awad, et al.: “The Moral Machine experiment”,Nature, 563 (7729), 59, 2018.

Please contact:

Mihály Héder, [email protected]://www.sztaki.hu/en/science/departments/hbit

Figure�1:�The�epistemic�constraints�of�the�autonomous�car�limit�the�scope�of�design-time�moral�investigations.


A Language for Graphs

of Interlinked Arguments

by Dimitra Zografistou, Giorgos Flouris, Theodore Patkos,and Dimitris Plexousakis (ICS-FORTH)

ArgQL is a high-level declarative language, aimed to

query data which are structured as a graph of

interconnected arguments. It provides specially designed

constructs and terminology that generate queries

relevant to the domain of interest that are both easy to

express and understand.

The recent advances in the technologies of Web 2.0 changedthe role of its users from passive information consumers toactive creators of digital content. Web became a universalterrain, wherein humans accommodate their inherent needfor communication and self-expression. From a scientificpoint of view, this new era was accompanied by numerousnew challenges. Navigation in dialogues and investigation ofthe informational requirements is one such challenge, whichconstitutes a pristine and until recently, almost untouchedarea. The process of human argumentation, in contrast, hasbeen a longstanding subject of theoretical studies.

Computational argumentation is a branch of AI and it offersmore accurate and realistic reasoning methods by transferringthe cognitive behaviour of people when arguing into its compu-tational models. An extensive overview in the area led us to theobservation that it also defines solid and discrete constructs thatstructure a dialogue. This observation motivated us to developArgQL (Argumentation Query Language) [1, 2], a novel, high-level declarative query language that will allow for the naviga-tion and information identification in a graph of interconnectedarguments, structured in the principles of argumentation.ArgQL constitutes an initial effort to understand the informa-tional and theoretical requirements during this process. Themost significant contribution lies in its potential to provide aquerying mechanism, focused on the internal structures ofarguments and their interactions, isolating the process fromtechnical details related to the traditional languages. Its need ishighlighted by the complexity of constructing SPARQLqueries, even for simple statements, like “How an argumentwith a given conclusion is attacked?” in the argumentationdomain. Instead, ArgQL generates quite elegant and represen-tative queries, easy to both express and understand.

ArgQL was designed to cover several predefined informa-tional requirements, which can be categorised as follows:• Individual arguments identification: We provide features

that allow to add constraints to the argument’s internalstructure, based on particular values.

• Correlated arguments identification: ArgQL also allowsconstraints to be expressed on an argument’s content withregard to other arguments, such as: search for pairs ofarguments with commonalities in their content.

• Argument relations extraction: ArgQL offers built-in key-words that allow express restrictions to be expressed aboutthe relations between arguments.

• Dialogue traversing and sub-dialogues identification:Expressions used for navigating across the relationbetween arguments are also provided.

In Figure 1, we show an example of the target data. In the leftpart, we show the data structures in the lowest level, con-sisting of arguments and two types of relations betweenpropositions, conflicts and equivalences, while the rightdepicts their abstract view, in which data form a graph ofinterlinked arguments.

Figure 2 shows two examples of ArgQL, along with an intu-itive description of the requirements captured by each.

As a first step, we formally define a data model based on theprevailing concepts in the area of computational argumenta-tion. Afterwards, we define the language specifications interms of its syntax, as well as its formal semantics that showhow the different keywords and expressions are evaluatedagainst the proposed data model. For query execution, we pro-pose a methodology to translate ArgQL into already well-known storage schemes and in particular the RDF/SPARQLlanguage. The methodology includes the mapping between thedata models and the translation between the query languages.The correctness of the translation has been formally proven.We have implemented ArgQL and have also developed anendpoint, wherein queries can be executed against realdatasets. The performance of the translation is experimentallyevaluated on these datasets. Despite its theoretical correctness,the proposed translation revealed some issues at implementa-tion time, which concerned particular query cases. To addressthose issues, we suggest a set of optimisations, which result inshorter and, therefore, more effective queries.

References:

[1] D. Zografistou, G. Flouris, D. Plexousakis: “Argql: Adeclarative language for querying argumentative dia-logues”, in International Joint Conference on Rules andReasoning (pp. 230-237), Springer, 2017.

[2] D. Zografistou, et al.: “Implementing the ArgQLQuery”, Computational Models of Argument, Proc. ofCOMMA 2018, 305, p.241, 2018.

Please contact:

Dimitra Zografistou, ICS-FORTH, Greece, +30 2810 391683, [email protected]


Figure�1:�Example�of�target�data.

Figure�2:�Two�examples�of�ArgQL.


• Deletion: An important aspect of the GDPR is informa-tional self-determination: This includes the right to subse-quent withdrawal of consent and, ultimately, the right todata deletion. Processes must accomodate this right; there-fore, methods for the unrecoverable deletion of data fromcomplex data-processing systems are needed, a fact whichstands in direct opposition to the design criteria that havebeen employed since the advent of computing. Deletion asa whole is also a problem with respect to explainabilityand retraceability, thus opening up a substantial newresearch field on these topics.

• Data security: The GDPR not only prescribes privacy pro-tection of personal data, but also data security. Varioussolutions already exist for this; the challenge does not liein the technical research but in the concrete applicationand budgetary framework conditions.

• Informed consent: Consent is another essential componentof the GDPR. Henceforth, users will have to be askedmuch more clearly for their consent to using their data formore explicitly specified purposes. The academic andlegal worlds have already made many suggestions in thisarea, so, in principle, this problem can be consideredsolved.

• Fingerprinting: Often data is willingly shared with otherinstitutions, especially in the area of data-driven research,where specialised expertise by external experts and/or sci-entists is required. When several external data recipientsare in the possession of data, it is important to be able todetect a leaking party beyond doubt. Fingerprinting pro-vides this feature, but most mechanisms currently in useare unable to detect a leak based on just a single leakedrecord.

Within the framework of the Big Data Analytics project,methods for solving these challenges will be analysed, with

the aim of coming up with practical solutions, i. e., the prob-lems will be defined from the point of view of concrete usersinstead of using generic machine learning algorithms ongeneric research data. In our testbed, we will implement sev-eral anonymisation concepts based on k-anonymity andrelated criteria, as well as several generalisation paradigms(full domain, subtree, sibling) combined with suppression.Our partners from various data-driven business areas (e. g.medical, IT security, telecommunications) provide completeuse cases, combining real-life data with the respectivemachine learning workflows. These use cases will be sub-jected to the different anonymisation strategies, thusallowing the actual distortion introduced by them to be meas-

49

Distortion in Real-World

Analytic Processes

by Peter Kieseberg (St. Pölten University of AppliedSciences), Lukas Klausner (St. Pölten University of AppliedSciences) and Andreas Holzinger (Medical University Graz).

In discussions on the General Data Protection Regulation

(GDPR), anonymisation and deletion are frequently

mentioned as suitable technical and organisational

methods (TOMs) for privacy protection. The major

problem of distortion in machine learning environments,

as well as related issues with respect to privacy, are

rarely mentioned. The Big Data Analytics project

addresses these issues.

People are becoming increasingly aware of the issue of dataprotection, a concern that is in part driven by the use of per-sonal information in novel business models. The essentiallegal basis for considering the protection of personal data inEurope has been created in recent years with the GeneralData Protection Regulation (GDPR). The data protectionefforts are confronted with a multitude of interests inresearch [1] and business [2], which are based on the provi-sion of often sensitive and personal data. We analysed themajor challenges in the practical handling of such data pro-cessing applications, in particular the challenges they pose toinnovation and growth of domestic companies, with partic-ular emphasis on the following areas (see also summary inFigure 1):• Anonymisation: For data processing applications to be

usable, it is essential, particularly in the area of machinelearning, that the obtained results are of high quality. Clas-sical anonymisation methods general-ly distort the results quite strongly[3]. Mere pseudonymisation, typical-ly used up to now as a replacement,can no longer be used as a privacyprotection measure, since the GDPRexplicitly stipulates that these meth-ods are not sufficiently effective. Atpresent, however, there is no large-scale study on these effects whichconsiders different types of informa-tion. Also, the approaches to mitigatethis distortion are currently still most-ly proofs of concept and purely aca-demic. Concrete methods are needed to reduce this distor-tion and to deal with the resulting effects.

• Transparency: The GDPR prescribes transparency in dataprocessing, i. e., the fact that a data subject has the right toreceive information about the data stored about them atany time, and to know how it is used. At present, no prac-tical methods exist to create this transparency whilstavoiding possible data leaks. Furthermore, the commonlyused mechanisms currently in use are not designed toensure transparency in the context of complex evaluationsusing machine learning algorithms. Guaranteeing trans-parency is also an important prerequisite for the deletionof personal information as well as for ensuring responsi-bility and reproducibility.

Figure�1:�Important�aspects�of�data�science�affected�by�the�GDPR.

EUROPEAN COMPUTER SCIENCE SUMMIT15th

ECSS 2019RRoommee - Italy

2288--3300 OCTOBER

ECSS is the forum to debate the trends and issues that

impact the future of Informatics in Europe.

Join leading decision makers and hear from renowned speakers about the challenges and opportunities for

Informatics research and education in an increasingly interconnected world.

KKeeyynnoottee SSppeeaakkeerrss::

! TToonnyy BBeellppaaeemmee, Ghent University

! VViirrggiinniiaa DDiiggnnuumm, Umeå University! MMiirreeiillllee HHiillddeebbrraannddtt, Vrije Universiteit Brussel

! PPaaoollaa IInnvveerraarrddii, University of L‘Aquila

! BBeerrnndd SSttaahhll, De Montfort University

EEvveennttss::

! Leaders Workshop ! WIRE Workshop! Keynotes and Panel on Social Responsibility of

Informatics! Informatics Europe - ERCIM Awards Session ! Informatics Europe Sessions! Talent Gap Workshop

CCoo--llooccaatteedd eevveennttss::

! EERRCCIIMM FFaallll MMeeeettiinnggss aanndd 3300tthh AAnnnniivveerrssaarryy CCeelleebbrraattiioonn

www.informatics-europe.org/ecss

Organized by

#ECSS_2019

In partnership with

! Pontifical Lateran University! “Sapienza” University of Rome! University of Rome “Tor Vergata”! University of Rome “Tre”

20

POREU15th

SECSemoR - ylaIt

82 - 03 REOBTOC

9

TUPMOCNEA

1 20CE

pmi

ioJesp

ofnIetin

t

E CENICSERTs

ehtetabedotmurofehts iSS

nis citamrofnIfoerutufehttcap

dnas rekamnosiicedgnidaelnnas egnellahcehttuobas rekae

oitacudednahcraseers citamroldrowdetcennocre

citamrofniwww. -

t

TIMMUSE s

ahts essuidnas dnert

.eporuEn

denwonermorfraehrofs eitinutroppod

no ylgnsiaercninani

sec/groe.poreu9102_SSC#E

etin

eK

!!!!!

vE

.ldrowdetcennocre

srekaepSetonye :

ynoT emeapleB , sirevinUtnehGainigriV mungiD , åeUm revinU

tdnarbedliHellieriM vinUejiVr,aloaP idrarevnI LfoytsirevinU,

lhatSdnreB vinUtroftnoMeD,

:stnev

dLe hkWo

ytsi

ytsirlsseurBtietsirev

aliuqA‘L

ytsire

dezinagOr

srentrapIn

! aretaLlacifitnoP! ” azneipa“S vinU! oRfoytsirevinU! ytsirevinU omRof

!!!

!!!

oC

!

yb

hitwiph

ytsirevinUnafoytsirev emRo

emo or“T atagrVe ”eom ”er“T

sredaLe pohskrWopohskrWoERWI

laicoSnolenaPdnasetoyneKscitamrofIn

eporuEscitamrofIn - AMICERsnoisseSeporuEscitamrofIn

ophskorWpaGtnelTa

o- detacol :stneve

MICRE sgniteeMllaF ht03dna

foy tilibisnopseR

n oisseSsdarwA

noitarbeleCyrasrevinnA

New Project

GAtEKEEPER - Smart Living

Homes for People at Health

and Social Risks

European citizens are living longer and this is putting pres-sure on European Healthcare to detect conditions and risksearly, and to manage them properly. There are multiple effortsto address this with a common focus on digital transforma-tion. ERCIM will be part of the 42 month duration GATE-KEEPER EU H2020 project to create an open source hub forconnecting healthcare providers, businesses, entrepreneurs,elderly citizens and the communities they live in. The aim isto provide a framework for creating and exploiting combineddigital solutions for personalised early detection and interven-tions, and to demonstrate the value across eight regional com-munities from seven EU member states. The technical under-pinnings include the Web of Things for integrating a varietyof sensor technologies along with FHIR and SAREF for e-health records and semantic models. W3C/ERCIM will con-tribute its experience with developing Web technology stan-dards in relation to the Web of Things and semantic interoper-ability, and will seek to exploit the work on open markets ofservices as the basis for future standardisation.

Link: http://www.gatekeeper-project.eu/Please contact: Dave Raggett, W3C, [email protected].

50 ERCIM NEWS 118 July 2019

ured. This distortion will be evaluated with respect to the usecases’ quality requirements.

In summary, this project addresses the question of whetherthe distortion introduced through anonymisation hampersmachine learning in various application domains, and whichtechniques seem to be most promising for distortion-reducedprivacy-aware machine learning.

References:

[1] O. Nyrén; M. Stenbeck, H. Grönberg: “The EuropeanParliament proposal for the new EU General Data Pro-tection Regulation May Severely Restrict EuropeanEpidemiological Research”, in: European Journal ofEpidemiology 29 (4), S. 227–230, 2014. DOI: 10.1007/s10654-014-9909-0, https://kwz.me/hyj

[2] S. Ciriani: “The Economic Impact of the EuropeanReform of Data Protection”, in Communications &Strategies 97 (1), 41–58, SSRN: 26740102015.

[3] B. Malle, et al.: “The right to be forgotten: towardsmachine learning on perturbed knowledge bases”, inProc. of ARES 2016, pp. 251-266, Springer, 2016.DOI: 10.1007/978-3-319-45507-5_17

Please contact:

Peter KiesebergSt. Pölten University of Applied Sciences, [email protected]


HoRIZoN 2020 Project Management

A European project can be a richly rewarding tool for pushing your research orinnovation activities to the state-of-the-art and beyond. Through ERCIM, ourmember institutes have participated in more than 90 projects funded by theEuropean Commission in the ICT domain, by carrying out joint research activitieswhile the ERCIM Office successfully manages the complexity of the projectadministration, finances and outreach. The ERCIM Office has recognizedexpertise in a full range of services, including identification of funding opportuni-ties, recruitment of project partners, proposal writing and project negotiation, con-tractual and consortium management, communications and systems support,organization of events, from team meetings to large-scale workshops and confer-ences, support for the dissemination of results.

How does it work in practice?

Contact the ERCIM Office to present your project idea and a panel of experts willreview your idea and provide recommendations. If the ERCIM Office expressesits interest to participate, it will assist the project consortium either as projectcoordinator or project partner.

Please contact:

Peter Kunz, ERCIM Office, [email protected]

Call for Proposals

Dagstuhl Seminars

and Perspectives

Workshops

Schloss Dagstuhl – Leibniz-Zentrum für

Informatik is accepting proposals for

scientific seminars/workshops in all

areas of computer science, in particu-

lar also in connection with other fields.

If accepted the event will be hosted inthe seclusion of Dagstuhl’s well known,own, dedicated facilities in Wadern onthe western fringe of Germany.Moreover, the Dagstuhl office willassume most of the organisational/administrative work, and the Dagstuhlscientific staff will support the organ-izers in preparing, running, and docu-menting the event. Thanks to subsidiesthe costs are very low for participants.

Dagstuhl events are typically proposedby a group of three to four outstandingresearchers of different affiliations. Thisorganizer team should represent a rangeof research communities and reflectDagstuhl’s international orientation.More information, in particular, detailsabout event form and setup as well as theproposal form and the proposing processcan be found on

https://www.dagstuhl.de/dsproposal

Schloss Dagstuhl – Leibniz-Zentrum fürInformatik is funded by the German fed-eral and state government. It pursues amission of furthering world classresearch in computer science by facili-tating communication and interactionbetween researchers.

Important Dates• Proposal submission: October 15 to

November 1, 2019• Notification: February 2020• Seminar dates: Between September

2020 and August 2021 (tentative).

In Memory of Cor baayen

It is with great sadness that welearned that Cor Baayen, first presi-dent and co-founder of ERCIM,passed away on Wednesday, 22 May2019. Baayen served as ERCIMpresident from 1991 to 1994. Hewas scientific director of CWI from1980 to 1994. During this period, heplayed a key role in shaping com-puter science as a distinct scientificfield. Under his leadership CWI

transformed from a mathematical institute to a centre of expertise for both mathe-matics and computer science. Together with Alain Bensoussan from Inria andGerhard Seegmüller from the former GMD, he founded ERCIM in 1989 with theaim of building a European scientific community in information technology.

Baayen started his first tenure at CWI in 1959, when it was still namedMathematisch Centrum (MC). He was appointed leader of the pure mathematicsgroup at MC in 1965, as well as professor of mathematics a the Vrije Universiteitin Amsterdam. When Baayen became scientific director in 1980, he immediatelyhad to deal with diminishing funds for academic research. One of the most notablestrategies to secure MC’s future, was to incorporate the institute in the first Dutchnational ICT funding scheme. This fitted perfectly with a broadening of the insti-tute’s focus towards computer science. A milestone in this transformation is therenaming of the institute to CWI (Centrum voor Wiskunde en Informatica, com-prising informatics) in 1983.

Cor Baayen will be remembered as one of the founders of ERCIM. The consor-tium grew out to become Europe’s most ambitious association in this field, withcurrently 16 member institutes. ERCIM honours Baayen’s legacy with its annualCor Baayen Young Researcher Award for promising young researchers in com-puter science or applied mathematics.

51ERCIM NEWS 118 July 2019

Photo: CWI

ERCIM is the European Host of the World Wide Web Consortium.

Institut National de Recherche en Informatique et en AutomatiqueB.P. 105, F-78153 Le Chesnay, Francewww.inria.fr

VTT Technical Research Centre of Finland LtdPO Box 1000FIN-02044 VTT, Finlandwww.vttresearch.com

SBA Research gGmbHFavoritenstraße 16, 1040 Wienwww.sba-research.org/

Norwegian University of Science and Technology Faculty of Information Technology, Mathematics and Electri-cal Engineering, N 7491 Trondheim, Norwayhttp://www.ntnu.no/

Universty of WarsawFaculty of Mathematics, Informatics and MechanicsBanacha 2, 02-097 Warsaw, Polandwww.mimuw.edu.pl/

Consiglio Nazionale delle RicercheArea della Ricerca CNR di PisaVia G. Moruzzi 1, 56124 Pisa, Italywww.iit.cnr.it

Centrum Wiskunde & InformaticaScience Park 123, NL-1098 XG Amsterdam, The Netherlandswww.cwi.nl

Foundation for Research and Technology – HellasInstitute of Computer ScienceP.O. Box 1385, GR-71110 Heraklion, Crete, Greecewww.ics.forth.grFORTH

Fonds National de la Recherche6, rue Antoine de Saint-Exupéry, B.P. 1777L-1017 Luxembourg-Kirchbergwww.fnr.lu

Fraunhofer ICT GroupAnna-Louisa-Karsch-Str. 210178 Berlin, Germanywww.iuk.fraunhofer.de

RISE SICSBox 1263, SE-164 29 Kista, Swedenhttp://www.sics.se/

Magyar Tudományos AkadémiaSzámítástechnikai és Automatizálási Kutató IntézetP.O. Box 63, H-1518 Budapest, Hungarywww.sztaki.hu/

TNOPO Box 968292509 JE DEN HAAGwww.tno.nl

University of CyprusP.O. Box 205371678 Nicosia, Cypruswww.cs.ucy.ac.cy/

ERCIM – the European Research Consortium for Informatics and Mathematics is an organisa-

tion dedicated to the advancement of European research and development in information tech-

nology and applied mathematics. Its member institutions aim to foster collaborative work with-

in the European research community and to increase co-operation with European industry.

INESCc/o INESC Porto, Campus da FEUP, Rua Dr. Roberto Frias, nº 378,4200-465 Porto, Portugal www.inesc.pt

I.S.I. – Industrial Systems InstitutePatras Science Park buildingPlatani, Patras, Greece, GR-26504 www.isi.gr

Subscribe to ERCIM News and order back copies at https://ercim-news.ercim.eu/

Documents

Special theme: .,.8&1 *&18- - ERCIM News · 35 SmartWork: Supporting Active and Healthy Ageing at Work for Office Workers by Otilia Kocsis, Nikos Fakotakis and Konstantinos Moustakas