Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Feature Extraction for Modeling Patients’ Outcomes:
an Application to Readmissions in ICUs
Rita Domingues Viegas
Thesis to obtain the Master of Science Degree in
Mechanical Engineering
Supervisor: Dr. Susana Margarida da Silva Vieira
Examination Committee
Chairperson: Prof. João Rogério Caldas PintoSupervisor: Dr. Susana Margarida da Silva Vieira
Member of the Committee: Prof. João Miguel da Costa Sousa
June 2015
To my family, the best teachers,
to the friends who are family,
and to the teachers who became friends.
i
ii
Abstract
The aim of this dissertation is to compare the performance of different approaches in the prediction of
patients’ readmissions in intensive care units (ICUs). To do so, fuzzy modeling techniques are applied to
data comprising patient demographics and measurements of physiological variables collected through
the patients’ hospitalization period.
Initially, patients are divided in two groups, readmitted and not readmitted, where the former corre-
sponds to patients readmitted to the ICU within 24 to 72 hours after discharge. Each patient is then
evaluated with a sampling time of 24 hours, so that new samples relative to each one of the hospital-
ization days can be obtained, which is necessary to develop models capable of predicting the risk of
readmission at any stage of the hospitalization.
In a first approach, demographic variables and the current values of each physiological variable
are used directly to develop fuzzy models. Following, a feature extraction approach is implemented
where additional information contained in the time series variables is included, as the maximum and
minimum values measured until the current evaluation moment. Finally, the feature extraction approach
is combined with a multimodel approach. Multimodels are hence implemented where different feature
selection and decision criteria are considered.
Overall, an increase in performance was achieved with the implementation of the proposed ap-
proaches, having the best model been obtained with the feature extraction and multimodel approach.
Keywords: Feature Extraction, Multimodel Approach, ICU, Readmissions, Fuzzy Clustering
iii
iv
Resumo
A presente dissertacao tem como objetivo a comparacao do desempenho de diferentes abordagens
na previsao de readmissoes de doentes em unidades de cuidados intensivos (UCIs). Para tal, sao
aplicadas tecnicas de modelacao fuzzy a dados demograficos e a medicoes de variaveis fisiologicas
recolhidas durante a hospitalizacao dos doentes.
Inicialmente, os doentes sao divididos em readmitidos e nao readmitidos, onde os primeiros
correspondem aos doentes readmitidos na UCI 24 a 72 horas apos a alta. Cada doente e avaliado
com um tempo de amostragem de 24 horas, de forma a obter novas amostras relativas a cada dia
da hospitalizacao, necessarias para o desenvolvimento de modelos capazes de prever o risco de
readmissao em qualquer momento da hospitalizacao.
Numa primeira abordagem, sao utilizadas as variaveis demograficas e os valores atuais de cada
variavel fisiologica diretamente para construir modelos fuzzy. Em seguida, e implementada uma abor-
dagem de extracao de variaveis, onde sao incluıdas informacoes adicionais contidas nas series tempo-
rais, como os valores maximos e mınimos medidos ate ao momento da avaliacao atual. Finalmente, a
abordagem de extracao de variaveis e combinada com uma abordagem de multimodelos. Deste modo
sao implementados multimodelos, onde diferentes criterios de selecao de variaveis e de decisao sao
testados.
De uma forma global, verificou-se um aumento no desempenho com a implementacao das abor-
dagens propostas, tendo a abordagem multimodelo com extracao de variaveis resultado no melhor
modelo.
Palavras-chave: Extracao de Variaveis, Abordagem Multimodelo, UCI, Readmissoes, Fuzzy
Clustering
v
vi
Acknowledgments
At first, I must address a special thank you to my supervisor, Professor Susana Vieira, for her constant
support and guidance through the development of this thesis, her knowledge on this area of research
has truly inspired me.
At second, I must thank Catia Salgado for her never-ending availability to discuss ideas and for
providing me so many thoughtful suggestions and words of encouragement through the work.
To my remaining lab partners, Lucia Cruz, Marta Ferreira, Carlos Azevedo, Hugo Proenca and Hugo
Santos, whose company made this such a pleasant task, I must also address a big thank you for all the
brainstorming and cheering up through the work.
Also to all the remaining friends I’ve met through my journey in IST, specially Joana Correia, Beatriz
Lopez, Ana Angelino, Zita Carreira, Tomas Hipolito, Diogo Ruivo, Joao Caldeira and David Santos, who
accompanied me in countless moments, heard my complains in difficult times and always believed in
me and cheered me up, I must address the biggest thank you. These last years would have been much
more difficult without them.
Also to my family, without whom I wouldn’t have made it to this point, I must address a huge thank
you for all the unconditional love and support, for being such inspirational role models and for all the
priceless values they have taught me.
At last, I would like to take the opportunity to thank all the amazing friends, who even often at a
distance have supported me for endless years and to the great teachers I had the opportunity of meeting
through my academic path for believing in me and having inspired me to be better.
vii
viii
Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Resumo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii
Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Readmissions in ICUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Knowledge Discovery 5
2.1 Knowledge Discovery in Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4.1 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4.2 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4.2.1 Sequential Forward Selection . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Data Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.5.1 Model Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.6 Evaluation / Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Statistical Analysis of the Readmissions Source Data 13
3.1 Readmissions Source Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.2 Age and Gender . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 ICU Length of Stay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Number of Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.5 Elapsed time between admission and first measurement . . . . . . . . . . . . . . . . . . . 18
ix
4 Readmissions Data Processing 23
4.1 Data Acquisiton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3 Output Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4 Prime Features Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.5 Extracted Features Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.6 Data Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5 Modeling 29
5.1 Fuzzy Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.1.1 Takagi-Sugeno Fuzzy Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2.1 Fuzzy C-Means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.2.2 Gustafson-Kessel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3 Multimodel Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.3.1 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.3.2 Decision Criterion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6 Results 37
6.1 Daily Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
6.1.1 Single Model with Prime Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
6.1.2 Single Model with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.1.3 Multimodel with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.1.3.1 Areas Under the Sensitivity and Specificity Curves . . . . . . . . . . . . . 41
6.1.3.2 Sensitivity and Specificity at the Intersection Threshold . . . . . . . . . . 42
6.1.3.3 Sensitivity and Specificity Close to the Intersection Threshold . . . . . . . 43
6.1.4 Results Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.1.5 Model Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.2 Discharge Day Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.2.1 Single Model with Prime Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.2.2 Single Model with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.2.3 Multimodel with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.2.3.1 Areas Under Sensitivity and Specificity Curves . . . . . . . . . . . . . . . 53
6.2.3.2 Sensitivity and Specificity at the Intersection Threshold . . . . . . . . . . 54
6.2.3.3 Sensitivity and Specificity Close to the Intersection Threshold . . . . . . . 55
6.2.4 Results Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7 Conclusions 59
7.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
x
Bibliography 67
Appendix A Daily Evaluation A1
A.1 Single Model with Prime Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A1
A.2 Single Model with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A3
A.3 Multimodel with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A5
Appendix B Discharge Day Evaluation B1
B.1 Single Model with Prime Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B1
B.2 Single Model with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B2
B.3 Multimodel with Extracted Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B3
xi
xii
List of Figures
2.1 Steps of the KDD procedure, adapted from [7] . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Schematic representation of the sequential forward selection process. . . . . . . . . . . . 10
2.3 Different ROC curves with the corresponding AUC value. . . . . . . . . . . . . . . . . . . 12
3.1 Age distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2 Age and gender distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 ICU length of stay. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.4 Heart rate measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5 Temperature measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.6 Platelets measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.7 NBP mean measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.8 SpO2 measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.9 Lactic acid measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.10 Creatinine measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.11 Troponin measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.12 Elapsed time between admission and first heart rate measurement. . . . . . . . . . . . . 19
3.13 Elapsed time between admission and first Temperature measurement. . . . . . . . . . . . 19
3.14 Elapsed time between admission and first platelets measurement. . . . . . . . . . . . . . 19
3.15 Elapsed time between admission and first NBP mean measurement. . . . . . . . . . . . . 20
3.16 Elapsed time between admission and first SpO2 measurement. . . . . . . . . . . . . . . . 20
3.17 Elapsed time between admission and first lactic acid (0.5-2.0) measurement. . . . . . . . 20
3.18 Elapsed time between admission and first creatinine (0-1.3) measurement. . . . . . . . . 21
3.19 Elapsed time between admission and first troponin measurement. . . . . . . . . . . . . . 21
4.1 Scheme of the data acquisition and data preprocessing steps. . . . . . . . . . . . . . . . 25
4.2 Output evolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3 Scheme of samples distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.4 Features acquired from the time series variables. . . . . . . . . . . . . . . . . . . . . . . . 27
5.1 Configuration of fuzzy systems configuration, adapted from [44] . . . . . . . . . . . . . . . 30
5.2 Multimodel configuration with a posteriori decision, adapted from [25, 26] . . . . . . . . . 34
xiii
6.1 AUC, accuracy, sensitivity and specificity obtained with the different approaches imple-
mented. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6.2 Real and predicted outcomes of the not readmitted patient with ID number 30062. . . . . 48
6.3 Real and predicted outcomes of the not readmitted patient with ID number 21871. . . . . 48
6.4 Real and predicted outcomes of the readmitted patient with ID number 31651. . . . . . . 48
6.5 Real and predicted outcomes of the not readmitted patient with ID number 20969. . . . . 49
6.6 Real and predicted outcomes of the not readmitted patient with ID number 27047. . . . . 49
6.7 Real and predicted outcomes of the readmitted patient with ID number 32170. . . . . . . 49
6.8 Real and predicted outcomes of the not readmitted patient with ID number 25862. . . . . 49
6.9 Real and predicted outcomes of the not readmitted patient with ID number 26159. . . . . 49
6.10 Real and predicted outcomes of the readmitted patient with ID number 26980. . . . . . . 50
6.11 AUC, accuracy, sensitivity and specificity obtained with the different approaches imple-
mented. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
xiv
List of Tables
2.1 Possible combinations of predicted and real outcomes. . . . . . . . . . . . . . . . . . . . . 11
3.1 List of initially considered variables from the MIMIC II database and corresponding total
number of measurements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.1 Final variables considered from the MIMIC II database. . . . . . . . . . . . . . . . . . . . 24
4.2 Physiological ranges of the variables considered from the MIMIC II database. . . . . . . . 24
6.1 Best combination of parameters obtained with prime features after 5-fold cross-validation. 38
6.2 Best subsets of features obtained by performing SFS with 5-fold cross-validation with
prime features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.3 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA subset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.4 Best combination of parameters obtained with extracted features after 5-fold cross-validation. 40
6.5 Best subsets of features obtained by performing SFS with 5-fold cross-validation with
extracted features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.6 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
6.7 Best subsets of features for the model highlighting sensitivity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the first set of feature selection
criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.8 Best subsets of features for the model highlighting specificity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the first set of feature selection
criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
6.9 Mean and standard deviation of the best results obtained after performing 5 × 5-fold
cross-validation in the MA dataset for the first set of feature selection criteria. . . . . . . . 42
6.10 Best subsets of features for the model highlighting sensitivity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the second set of feature
selection criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.11 Best subsets of features for the model highlighting specificity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the second set of feature
selection criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
xv
6.12 Mean and standard deviation of the best results obtained after performing 5 × 5-fold
cross-validation in the MA dataset for the second set of feature selection criteria. . . . . . 43
6.13 Best subsets of features for the model highlighting sensitivity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the third set of feature selec-
tion criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.14 Best subsets of features for the model highlighting specificity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the third set of feature selec-
tion criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.15 Mean and standard deviation of the best results obtained after performing 5 × 5-fold
cross-validation in the MA dataset for the third set of feature selection criteria. . . . . . . . 44
6.16 Mean and standard deviation of the AUC values obtained for the best multimodels devel-
oped. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.17 P-values between the best multimodels developed. . . . . . . . . . . . . . . . . . . . . . . 46
6.18 Performance comparison of different models. . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.19 Features selected for the sensitivity subset 1 and the specificity subset 3. . . . . . . . . . 48
6.20 Best combination of parameters obtained with prime features after 5-fold cross-validation. 50
6.21 Best subsets of features obtained by performing SFS with 5-fold cross-validation with
prime features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.22 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA subset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.23 Best combination of parameters obtained with extracted features after 5-fold cross-validation. 52
6.24 Best subsets of features obtained by performing SFS with 5-fold cross-validation with
extracted features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.25 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.26 Best subsets of features for the model highlighting sensitivity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the first set of feature selection
criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.27 Best subsets of features for the model highlighting specificity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the first set of feature selection
criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.28 Mean and standard deviation of the best results obtained after performing 5 × 5-fold
cross-validation in the MA dataset for the first set of feature selection criteria. . . . . . . . 54
6.29 Best subsets of features for the model highlighting sensitivity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the second set of feature
selection criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.30 Best subsets of features for the model highlighting specificity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the second set of feature
selection criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
xvi
6.31 Mean and standard deviation of the best results obtained after performing 5 × 5-fold
cross-validation in the MA dataset for the second set of feature selection criteria. . . . . . 55
6.32 Best subsets of features for the model highlighting sensitivity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the third set of feature selec-
tion criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.33 Best subsets of features for the model highlighting specificity, obtained by performing
SFS with 5-fold cross-validation with extracted features and the third set of feature selec-
tion criteria. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.34 Mean and standard deviation of the best results obtained after performing 5 × 5-fold
cross-validation in the MA dataset for the third set of feature selection criteria. . . . . . . . 56
A.1 Prime features acquired from the MIMIC II database. . . . . . . . . . . . . . . . . . . . . . A1
A.2 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the FS dataset with prime features, varying the number of clusters between
2 and 10 and the degree of fuzziness between 1.1 and 1.5. . . . . . . . . . . . . . . . . . A2
A.3 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the FS dataset with prime features, varying the number of clusters between
2 and 10 and the degree of fuzziness between 1.6 and 2. . . . . . . . . . . . . . . . . . . A2
A.4 Prime features obtained for each subset, after SFS with 5-fold cross-validation. . . . . . . A2
A.5 Extracted features acquired from the MIMIC II database. . . . . . . . . . . . . . . . . . . . A3
A.6 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the FS dataset with extracted features, varying the number of clusters be-
tween 2 and 10 and the degree of fuzziness between 1.1 and 1.5. . . . . . . . . . . . . . A4
A.7 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the FS dataset with extracted features, varying the number of clusters be-
tween 2 and 10 and the degree of fuzziness between 1.6 and 2. . . . . . . . . . . . . . . A4
A.8 Extracted features obtained for each subset, after SFS with 5-fold cross-validation. . . . . A4
A.9 Extracted features obtained for each subset, by performing SFS with the first set of feature
selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . . . . . A5
A.10 Extracted features obtained for each subset, by performing SFS with the second set of
feature selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . A5
A.11 Extracted features obtained for each subset, by performing SFS with the third set of fea-
ture selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . . . A5
A.12 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA dataset for the first set of feature selection criteria. . . . . . . . . . . . A6
A.13 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA dataset for the second set of feature selection criteria. . . . . . . . . . A7
A.14 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA dataset for the third set of feature selection criteria. . . . . . . . . . . A8
xvii
B.1 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the FS dataset with extracted features, varying the number of clusters be-
tween 2 and 10 and the degree of fuzziness between 1.1 and 1.5. . . . . . . . . . . . . . B1
B.2 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the FS dataset with prime features, varying the number of clusters between
2 and 10 and the degree of fuzziness between 1.6 and 2. . . . . . . . . . . . . . . . . . . B2
B.3 Prime features obtained for each subset, after SFS with 5-fold cross-validation. . . . . . . B2
B.4 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the FS dataset with extracted features, varying the number of clusters be-
tween 2 and 10 and the degree of fuzziness between 1.1 and 1.5. . . . . . . . . . . . . . B2
B.5 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the FS dataset with extracted features, varying the number of clusters be-
tween 2 and 10 and the degree of fuzziness between 1.6 and 2. . . . . . . . . . . . . . . B3
B.6 Extracted features obtained for each subset, after SFS with 5-fold cross-validation. . . . . B3
B.7 Extracted features obtained for each subset, by performing SFS with the first set of feature
selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . . . . . B3
B.8 Extracted features obtained for each subset, by performing SFS with the second set of
feature selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . B4
B.9 Extracted features obtained for each subset, by performing SFS with the third set of fea-
ture selection criteria, with 5-fold cross-validation. . . . . . . . . . . . . . . . . . . . . . . . B4
B.10 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA dataset for the first set of feature selection criteria. . . . . . . . . . . . B5
B.11 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA dataset for the second set of feature selection criteria. . . . . . . . . . B6
B.12 Mean and standard deviation of the results obtained after performing 5 × 5-fold cross-
validation in the MA dataset for the third set of feature selection criteria. . . . . . . . . . . 7
xviii
Notation
Symbols
α Feature selection threshold
βj Degree of activation of the jth rule
δ Final multimodel threshold
µAjn(x) Membership function of Ajn
µij Membership degree of sample j to the ith cluster
Ai Positive definite symmetric matrix
Ajn Fuzzy set of the jth rule and the nth feature
aj Parameter vector of rule Rj
bj Scalar offset of rule Rj
cs Sensitivity feature selection performance criterion
csp Specificity feature selection performance criterion
dij Distance from sample j to the ith cluster prototype
h Height
J Number of rules
Jm Objective Function
K Number of cross-validation folds
m Degree of fuzziness
N Total number of features
Ns Number of samples
nc Number of clusters
Pi Covariance matrix of cluster i
q Total number of features
Rj Fuzzy rule
sα Sensitivity value at threshold α
spα Specificity value at threshold α
t Threshold
tadm Number of days since admission in the ICU
tmax Time elapsed between admission and maximum value recorded until the evaluation point
tmin Time elapsed between admission and minimum value recorded until the evaluation point
U Partition matrix
V Prototypes matrix
vf Value of the first recorded measurement
vi Prototype of the ith cluster
vl Value of the last recorded measurement
xix
vm Mean of all values recorded until the evaluation point
vmax Maximum value recorded until the evaluation point
vmin Minimum value recorded until the evaluation point
w Weight
wβ Minor weighting factor of the model selection criterion
wφ Major weighting factor of the model selection criterion
wi Weighting factor of model i
X Dataset
x Sample input vector
xfmaxMaximum value for the f th feature
xfmin Minimum value for the f th feature
xif Prime value of sample i for feature f
xifnormNormalized value of sample i for feature f
y Sample output
yδ Multimodel final output
y1 Sensitivity model output
y2 Specificity model output
yc Continuous output
yj Consequent function of the jth rule
xx
Acronyms
ACC Accuracy
APACHE II Acute Physiology and Chronic Health Evaluation
AUC Area Under the Curve
BMI Body Mass Index
FCM Fuzzy C-Means
FN False Negative
FP False Positive
FPR False Positive Rate
FS Feature Selection
GK Gustafson-Kessel
ICU Intensive Care Unit
KDD Knowledge Discovery in Databases
MA Model Assessment
MIMIC II Multi-Parameter Intelligent Monitoring for Intensive Care
NBP Non-Invasive Arterial Blood Pressure
PCA Principal Component Analysis
ROC Receiver-Operating Characteristic
SFS Sequential Forward Selection
SpO2 Peripheral Capillary Oxygen Saturation
TN True Negative
TNR True Negative Rate
TP True Positive
TPR True Positive Rate
TS Takagi-Sugeno
xxi
xxii
Chapter 1
Introduction
1.1 Motivation
The amount of data available is growing at an exponential rate. As of 2012, around 2.5 exabytes of
data were created each day, and that number is doubling nearly every 40 months [1].
Such huge amounts of information available lead to equally huge amounts of knowledge opportu-
nities [2], which results in an increasing interest towards the potentialities of data in a variety of fields.
From financial applications [3] to healthcare [4], storm detection [5] to marketing and sales [6], every-
where data is being studied and analyzed in an attempt to extract and infer as much novel and useful
information as possible about the complex systems that surround us.
However, it is known how quantity does not necessarily stand for quality. Knowing how to make
sense of the data available, how to separate the relevant from the non-relevant information or how to
spot erroneous information that can otherwise lead to misleading results, represents a huge and time
consuming challenge.
From the beginning of time humans have dealt with this challenge, trying to understand the world
around them and to extract knowledge from the information available, first in an empirical way and
later resorting to manual analysis. However, when the quantity of data available is as massive and
rapidly growing as nowadays, such outdated tools are not enough, resulting in an urgent need for a new
generation of computational theories and tools [7].
Towards this need, began in recent decades the development of a new area of knowledge and
research, data mining. This area attempts to make the most of the computational power at our disposal,
being defined as the process of discovering patterns in data in an automatic or semi-automatic way [2].
One field where data mining can be extremely useful is the healthcare industry, an extremely data-
rich field where such technological tools are not yet as popularized as in other industries [8]. Neverthe-
less, the achievements that can be obtained through data mining techniques in this area are endless
and more and more that awareness is growing.
Resorting to it, and for example to the data mining technique of clustering where objects are classified
and organized in groups according to their similarities [9], specific patterns can be spotted in different
1
types of patients. Discovering such patterns, as well as the characteristics that make each patient belong
to a specific group, can then be used to a variety of purposes, as diagnosing different cancer types [10]
or identifying high-risk diabetes patients [4].
A topic in healthcare that can much benefit from data mining techniques is preventing patients from
being readmitted into intensive care units (ICUs), a highly relevant issue, as such patients have an
increased risk of death, longer length of stay and represent higher costs [11].
ICUs are subject to increasing pressures regarding the management of care and resources which
leads to the implementation of strategies seeking to rapidly free expensive beds. Such scenarios place
clinicians in difficult situations, as they must decide on the discharge of apparently stable patients to
make room for more urgent ones, risking the discharged patients to be readmitted in the short term [12].
The present study aims to help in this decision-making, by presenting a feature extraction and mul-
timodel approach combined with fuzzy modelling to predict patients’ readmissions into ICUs, based on
patient demographics and a set of physiological variables measured along the hospitalization period.
1.2 Readmissions in ICUs
A clinician’s decision on the discharge of a patient is always a delicate task. But when such deci-
sion takes place in ICUs where patients’ health conditions are extremely fragile and the case of early
discharge and consequent readmission can result in severe consequences, an even greater amount of
responsibility is at stake.
ICU readmissions are a commonly used quality measure but, despite it and the decades of research,
such events continue to occur at rates relatively unchanged over the years [13]. These readmissions
are of particular concern as readmitted patients have much worse prognoses than not readmitted ones.
Previous studies have shown how ICU readmissions can cause significant distress to patients and result
in considerable financial costs [14] or how in-hospital death rates can be two to ten times greater in
patients readmitted to ICUs than in not readmitted ones [15]. However, the amount of data collected
in ICUs has grown considerably in the last decades making it a particularly well-suited setting to imple-
ment data-driven systems to discover underlying patterns and relationships in intensive care data [16].
Considering this, several studies have been conducted in past years making use of the data collected in
ICUs to develop models to predict patients’ readmissions. Traditionally, such models have been based
on the clinical gold-standard method, the Acute Physiology and Chronic Health Evaluation II (APACHE
II) score [17], a severity of disease classification system [18]. Other studies have achieved slightly better
results applying logistic regressions to variables considered predictive of readmissions, as admission
diagnosis, treatment status, age, length of ICU stay or ICU mortality [19, 20].
More recently, different approaches were followed resorting to more complex methods. In [11], a
fuzzy model combined with tree search feature selection was developed based on physiological vari-
ables recorded in the last 24 hours before discharge, showing how it is possible to predict patients’
readmissions using a small number of variables. In [21], the application of fuzzy probabilistic systems
was proposed, having been achieved results comparable to the gold standard method, yet using a sig-
2
nificant lower amount of variables.
However, when it is intended to develop models based on records from physiological variables as-
sessed over time, some considerations must be taken. Such information, in the form of time series,
cannot be used straightforward as if they were constant features, like age or length of ICU stay.
To deal with it, several approaches can be followed. The arithmetic mean of each time series can
be computed, as in [11], or the procedure presented in [22, 23] can be followed where, additionally to
the arithmetic mean, other features as minimum, maximum, trends or first and last data points were
extracted from the time series variables to predict mortality in ICUs.
Finally, one additional approach has shown promising results in ICU applications, the multimodel ap-
proach. In [24], significantly better results were obtained using a multimodel scheme than a general one
to predict the necessity of vasopressors in sepsis patients and in [25] the same physiological variables
of [11] were used, but better results were obtained in the prediction of patients’ readmissions due to the
implementation of multimodels. Also in [26], the multimodel approach was addressed and considerably
well succeeded this time to predict the survival of ICU septic shock patients. In that work, a combined
strategy was implemented where initially separate models were developed, one aiming to maximize
sensitivity and the other to maximize specificity, being afterwards combined into a final output.
1.3 Contributions
The aim of this thesis is to combine two promising approaches to predict patients’ readmissions in
ICUs, feature extraction and multimodeling, and comparing them with the alternative approaches, the
use of prime features and single models.
A set of initial time series variables is considered based on a previous work [11] where such variables
were found to be the most predictive of ICU readmissions.
A feature extraction approach is then applied to the time series variables, aiming to retain more of
the information contained in the time series, in an alternative to the procedure presented in [11] where
solely the arithmetic mean of the values collected in the last 24 hours before discharge were used.
A multimodel is then developed based on the work of [25], but adding the concept of initially devel-
oping separate models, one highlighting the specificity and the other the sensitivity, as is suggested in
[26].
Furthermore, this thesis introduces one additional contribution to what was done in previous works
seeking to predict patients’ readmissions into ICUs. Instead of solely considering the last day of a
patients’ hospitalization, each patient is evaluated at different stages of his/her stay. In this way, more
information about the patients’ hospitalization is considered and a more suitable model for real world
applications is developed. Such suitability derives from the fact that in real world a clinician cannot know
a priori which is the last day of a patient’s hospitalization to decide on his discharge, making this a more
realistic approach.
3
1.4 Outline
Chapter 2 presents an overview of the main procedure followed as guideline for the generality of the
work, the Knowledge Discovery in Databases (KDD) procedure. The main steps that compose it are
described and a summary of methods that can be used in each step is presented.
In Chapter 3, the source readmissions data is presented and a statistical analysis is performed.
Patients are divided into two categories, readmitted and not readmitted, and the distributions of different
features per category are presented.
Chapter 4 presents the transformations applied to the initial data analyzed in Chapter 3 in order to
obtain the final datasets where to perform modeling. The topics of data acquisition, data preprocessing,
output definition and considered features are briefly explained and their application to the dataset is
presented.
In Chapter 5, the theoretical aspects of the modeling developed in the work are presented. Concepts
as fuzzy modeling, clustering algorithms and multimodel approach are introduced and their relevance to
the present work is presented.
In Chapter 6, the main results are summarized and compared. Results are presented for different
approaches, initially using a single model and prime features, afterwards a single model with extracted
features and, at last, a multimodel with extracted features.
Finally, in Chapter 7, the results of the different approaches are summarized and conclusions are
drawn. The benefits and limitations of each approach are discussed and directions for future research
are presented.
4
Chapter 2
Knowledge Discovery
This Chapter presents an overview of the main procedure followed throughout this thesis, the KDD
procedure. Its main constituent steps are described: selection, preprocessing, transformation, data
mining and evaluation.
2.1 Knowledge Discovery in Databases
The amount of data available is increasing at an enormous speed. In many fields huge amounts of
data are being collected, becoming essential to be able to deal with it and, specially, to know how to take
advantage of the knowledge opportunities that rise with it.
Considering the huge dimensions of this now available data, it becomes obvious how old techniques
based on human analysis have become obsolete and how the development of new procedures, re-
sorting to computational techniques and tools, has become mandatory, being a popular subject among
researchers. Towards this need, in 1996 the KDD procedure was firstly introduced [7], a procedure
commonly used in medical applications [27].
In [7], the KDD procedure is explained in detail and the set of steps that should to be taken when it is
intended to extract knowledge from raw data are presented. The KDD procedure has been defined as
a non-trivial process of identifying valid, novel, potentially useful and ultimately understandable patterns
from large collections of data.
According to [7], to extract knowledge from data the following main steps illustrated in Figure 2.1
should be taken:
- Selection;
- Preprocessing;
- Transformation;
- Data Mining;
- Evaluation / Interpretation.
5
Figure 2.1: Steps of the KDD procedure, adapted from [7] .
Nevertheless, it should be considered that even with an established procedure as this one, following
each of the steps is not always straight forward and the repetition of some of the steps in an iterative
and recursive way can be required, until satisfactory results are achieved.
In this work, the KDD procedure is followed as a guideline, being the description of each of its steps
introduced below.
2.2 Selection
Selection, also referred to as data acquisition [7], is the process of choosing the target data and
selecting the set of data samples and variables for the discovery process to focus. To be successful in
it, one should first understand the domain of application of the problem and clearly define the goal of
the KDD process, so that the more accurate and relevant target data can be defined. Once this has
been clarified, it involves the entire process of acquiring and storing the target data extracted from the
selected database, taking into account the inclusion and exclusion criteria defined for the data samples
and variables to be obtained.
In the present work, the selection step plays an important role, as one of the hypothesis of the
thesis regards selection of data and comparison with other approaches, where different data has been
selected.
2.3 Preprocessing
Data preprocessing is an essential step when dealing with virtually all real-world databases. Due
to their usual enormous dimensions and their likely origin from multiple and varied sources, these
databases are often incomplete, noisy, lacking attribute values, containing errors or outlier values and
inconsistent [2].
In the medical world, as in the majority of fields, there are countless potential sources of contami-
nation of the recorded data. It can be due to human errors, as misunderstandings or distractions, or to
machine ones, such as equipment malfunctions or lack of calibration, both contributing to the creation
6
of inaccurate data. Therefore, data preprocessing techniques, as handling missing data, noise and out-
liers, are essential to enable the improvement of the quality and reliability of the data, as well as the
reduction of the chances of biasing the knowledge extracted from it.
According to [2], to deal with missing values the following options can be chosen:
1. Ignore the tuple;
2. Input the missing values manually;
3. Use a global constant to input the missing values;
4. Use the attribute mean to input the missing values;
5. Use the attribute mean for all samples belonging to the same class as the given tuple;
6. Use the most probable value to input the missing values.
However, it should be noted that methods 3 to 6 can bias the data, as the imputed values may not be
the correct ones.
When faced with noise, a random error or variance in a measured variable, several methods, as
binning and regressions, can be used to remove it and thus smooth the data [2].
Binning methods work by sorting the data into a number of bins and replacing their values according
to the values inside each bin, using the mean, the median or the closest boundary value [2].
Alternatively, regressions can be used to smooth the data, as linear regressions where the best line
to fit two attributes is found so that one can be used to predict the other. In more complex cases the
same principle can be used, resorting to multi linear regressions instead where more than two attributes
can be involved [2].
Finally, to deal with data discrepancies or outliers, one should make use of the knowledge available
about the properties of the data, referred to as metadata [2]. This knowledge can include knowing the
domain, the data types or the acceptable values of each attribute, which can then be used to detect
values that do not respect it, outliers, and thus remove them. Different procedures can also be used, as
identifying as possible outliers values that are more than two standard deviations away from the mean
of a given attribute.
2.4 Transformation
Transformation and dimensionality reduction methods are extremely important, particularly when
dealing with high dimensional data sets, as these can increase substantially the size of the search
space and the chances of a data-mining algorithm finding erroneous patterns not valid in general [7].
Hence, the purpose of transformation and dimensionality reduction methods is obtaining a reduced yet
relevant set of features to represent the data, which can bring huge benefits both to the computational
time as to the performance of the resulting models.
7
There are two main categories in which these methods can be divided, feature extraction and feature
selection [28]. Feature extraction methods have the purpose of obtaining new features out of the original
variables while feature selection methods seek to select the most useful subset of features from an initial
set, while discarding the non-relevant ones.
2.4.1 Feature Extraction
Feature extraction techniques address the problem of finding the transformation of features contain-
ing the greatest amount of useful information, while attempting to achieve a reduced size set of features
[29]. Many techniques can be used to obtain such result, from complex linear and nonlinear statistical
procedures to simple mathematical operations.
One popular method is the principal component analysis (PCA) [30], a linear transformation that
enables dimensionality reduction, where a linear mapping of the data into a lower-dimensional space is
performed in a way that the variance of such data is maximized. This transformation results from the
eigenvectors corresponding to the largest eigenvalues of the covariance matrix of the data, which in turn
can be used to reconstruct a large fraction of the variance of the original data [28].
However, when dealing with time series variables, as is the case of the records of variables for each
patient along their stay, other methods can be more suitable to retain as much information as possible,
while diminishing substantially the dimensions of the dataset used. For example, for each time series,
the mean, the minimum, the maximum or the trend can be computed [22], as well as other properties
that can help to accurately represent the information contained in the initial time series.
2.4.2 Feature Selection
Feature selection can bring a variety of benefits, as diminishing the processing and storing require-
ments or facilitating the data visualization and understanding, by reducing the dimensionality of the
dataset [31]. However, it operates based on a sensitive compromise, as it is attempted to minimize the
number of selected features while retaining, as much as possible, the overall prediction information.
There are many methods used to perform feature selection but, in general, the main idea is finding a
subset of features that maximizes a specified criterion, while discarding the ones that present the worst
performances. In health care applications, as in other fields, this process brings also the advantage
of pointing out sets of features that might not have been initially chosen as the most relevant to the
prediction of an outcome by the human judgement.
According to [29], there are four main categories of feature selection methods: filters, wrappers,
hybrids and embedded.
Filter and wrapper methods both make use of search strategies to explore the space of all possible
combinations of features in order to select the best subsets. However, they differ substantially in the
evaluation criterion used. Filter methods use criteria that do not involve machine learning algorithms to
evaluate each feature or combination of features, as measurements of entropy, variance or relevance
indexes based on correlation coefficients among features. On the contrary, wrapper methods use the
8
performance of machine learning algorithms as criterion searching for the most predictive subset of
features, according to the chosen criterion.
In an attempt to take the most of the two previous methods, there are hybrid methods where first a
filter method is used to obtain a reduced set of features so that a wrapper method can then be used to
select the most relevant ones, according to the performance of a selected machine learning algorithm.
Finally, there are embedded methods that differ from the previous ones, as they incorporate simulta-
neously the generation of subsets of features and their evaluation in the model training process.
In terms of finding the candidate feature subsets to be evaluated, also different approaches can be
followed. Greedy search strategies are computational advantageous, since only a reduced area of the
search space is explored. They can be used in two ways, forward selection or backwards elimination.
In forward selection, variables are progressively incorporated into larger subsets, whereas in backward
elimination one starts with the set of all variables and progressively eliminates the least promising ones
[31]. Alternatively, more complex and computationally heavy search strategies can be used, as meta-
heuristics where the computational time maybe longer, but a greater amount of possibilities is considered
[32].
In this thesis, due to the complexity and large size of the dataset, a greedy search strategy is imple-
mented in a wrapper way, following the Sequential Forward Selection (SFS) approach.
2.4.2.1 Sequential Forward Selection
In a wrapper context, SFS works by building and evaluating models with increasingly complex sub-
sets of features, following the increase in performance of the developed models [33].
Initially, a performance criterion is chosen based on the objectives of the problem, which will allow
the comparison between the results obtained for each candidate feature subset. Once the criterion has
been selected, a model is built for each of the available features and its performance is evaluated. The
feature that returns the best performance score is kept and, in a second round, combined with each of
the remaining features. Once again, a model is built and evaluated for each of the new combinations
of features and only the best subset is kept. This process is repeated successively, always increasing
the number of selected features, as illustrated in Figure 2.2, until there is no increase in performance, in
which case the process is stopped, and the best subset of features found so far is kept.
This is a very popular method, having as advantages its simplicity and the possibility of an easy
graphical representation from which can easily be understood the performance of each candidate fea-
ture subset. However, it has the disadvantage of being susceptible to finding subsets of features corre-
sponding to local optima.
In the present work, this approach was chosen considering its acceptance and popularity in the
medical community, as well as the computational simplicity, extremely advantageous when dealing with
high dimensional datasets.
9
Figure 2.2: Schematic representation of the sequential forward selection process.
2.5 Data Mining
Data mining is a fundamental step of the KDD process, as it is the phase in which a primordial
knowledge starts being extracted from the data. It often involves repeating iteratively the use of one or
more data-mining methods in order to achieve the desired goals [7], which can be an extensive and time
consuming procedure.
Data mining can be used particularly to achieve two kinds of knowledge discovery goals: verification
and discovery. In verification it is intended to verify a user’s a priori hypothesis, being mostly used
common methods of traditional statistics. In discovery, the aim is to find new patterns from the data that
can be used to predict the future behaviour of a given system or to generate a description of the data, in
a more human-understandable form [7].
For description purposes, data mining involves fitting models to, or determining patterns from ob-
served data, so that knowledge can be inferred, like assigning data samples to classes as is the purpose
of the present work. Many methods can be used to obtain this result, but most of them are based on
tried and tested techniques from machine learning, pattern recognition and statistics, as classification,
clustering or regression [7].
The use of these methods can be associated to two major types of problems: unsupervised learning
and supervised learning [34].
In unsupervised learning the problem consists in discovering the underlying structure of the dataset,
if there is any. This means that the aim is finding if the data can be gathered in different groups according
to their similarities, and what these characteristics are that make objects similar within each group but
different among others. This is a complex problem, as there are no objective criteria equally accepted
to measure the results being the indication of the goodness of the results often subject to the user’s
estimate [34].
In supervised learning, on the contrary, each data sample is preassigned to a class label. The goal
is then to train a classifier to understand and model the relationships between the attributes of each
sample and the corresponding label so that if new data is introduced the model can, as accurately as
possible, assign the new samples to the correct label. This is the case of the present study where it is
10
intended to classify patients into the readmitted or not readmitted classes, by first training a model and
then testing it. It represents also a complex problem, but when compared to unsupervised learning has
the advantage of having several popularized objective criteria that can be used to evaluate the results,
as the ones presented in the following section.
2.5.1 Model Assessment
In supervised learning, several performance measures can be used to access the performance of a
given model. One of the most popular in classification problems is the area under the receiver-operating
characteristic (ROC) curve, the AUC [35], very common in the performance evaluation of classifiers in
medical problems [36].
To understand it, it is necessary to first be aware of how in the case of binary classifiers (as the
one being implemented in the present work) each sample can belong to one of the classes, positive
or negative and, similarly, the output of the classifier can belong to one of the two classes. Should be
noted though, that to obtain binary values from a classifier, generally with the values of 0 or 1, it is first
necessary to compare the initially continuous output with a value chosen a priori, the threshold, used as
a delimiter of the two classes.
Taking into account the combination of both real and predicted classes, four possible combinations
arise, presented in Table 2.1.
Table 2.1: Possible combinations of predicted and real outcomes.
Predicted ClassPositive Negative
Real ClassPositive True Positive (TP) False Negative (FN)
Negative False Positive (FP) True Negative (TN)
In this way, TP corresponds to samples correctly classified in the positive class, FP to samples incor-
rectly classified in the positive class, FN to samples incorrectly classified as belonging to the negative
class and TN to samples correctly classified in the negative class.
With this notation other important performance measurements can then be defined, as:
- Sensitivity, or true positive rate (TPR), that measures the quantity of positive samples correctly
classified as such:
Sensitivity = TPR =TP
TP + FN. (2.1)
- Specificity, or true negative rate (TNR), that measures the quantity of negative samples correctly
classified as such:
Specificity = TNR =TN
TN + FP. (2.2)
11
- Accuracy (ACC), that measures the proportion of correctly classifications over the total number of
observations:
Accuracy =TN + TP
TN + TP + FP + FN. (2.3)
Finally, the ROC curve can be introduced, as it is obtained by plotting the TPR, sensitivity, against
the false positive ratio (FPR), one minus the specificity, for varying thresholds between 0 and 1. The
AUC is then obtained by integrating this curve over all thresholds, such that AUC values between 0 to
1 can be obtained, where 0.5 corresponds to a random classifier and 1 to a perfect one. Values bellow
0.5 mean the classifier is performing better for the inverse problem, requiring an exchange of classes.
Figure 2.3 presents examples of different ROC curves and their corresponding AUC value.
AUC = 1
0,5 < AUC < 1
AUC = 0,5
FPR
TPR
0 0.2 0.4 0.6 0.8 1.0
1.0
0.8
0.6
0.4
0.2
0
Figure 2.3: Different ROC curves with the corresponding AUC value.
2.6 Evaluation / Interpretation
The last step of the KDD process is evaluation and interpretation. Even being the last, it is also a
crucial step, as it requires interpreting and evaluating critically the knowledge obtained, considering its
validity, impact, usefulness and novelty. At this point, it is also required to consider the present results
and to choose whether or not to return to any of the previous steps in order to achieve better or more
valid results.
In the present thesis, all of the mentioned steps of the KDD procedure are covered, since all are
necessary and vital to successfully extract knowledge from raw data. However, a special emphasis is
given to the steps of data acquisition, data transformation and data mining. In terms of data acquisition,
a novelty is introduced as the target data and prime variables selected differ from previous approaches
for the application problem. In data transformation, feature extraction is applied, which combined with a
multimodel approach implemented in the data mining step, makes the present a novel approach to the
problem of predicting patients’ readmissions in ICUs.
12
Chapter 3
Statistical Analysis of the
Readmissions Source Data
In this Chapter, the database used and the initial dataset considered are presented. In a first ap-
proach, a statistical analysis of this dataset is performed with the aim of better understanding the influ-
ence of different variables in the two categories of patients, readmitted and not readmitted to any ICU
within 24 to 72 hours after discharge.
3.1 Readmissions Source Data
To conduct the present work, an initial dataset was obtained from the Multi-Parameter Intelligent
Monitoring for Intensive Care (MIMIC II) database [37], a public research archive of data collected from
over 32,000 ICU patients. This large database has been de-identified by removal of all Protected Health
Information and contains data collected at the Beth Israel Deaconess Medical Center in Boston from
patients admitted to a variety of ICUs from 2001 to 2008. It comprises clinical information, as patients’
demographics, laboratory test results, vital signs recordings, fluid and medication records, charted pa-
rameters and free-text reports as nursing notes, imaging reports and discharge summaries.
In a first approach, the adult patients, that is, older than 15 year, that survived within one year of
discharge and that were admitted to the ICU for a minimum of 24 hours were selected from the original
database. Patients who returned to any ICU of the same medical center between 24 and 72 hours were
classified as readmitted and the ones that did not return to any ICU classified as not readmitted, resulting
in a set of 12,080 not readmitted patients and 775 readmitted ones, corresponding to 6.4 % of the total
of not readmitted patients. Being readmitted between 24 and 72 hours after discharged is considered an
early readmission [38]. The choice of 24 hours as the lower bound relies on the fact that in the MIMIC
II database patients readmitted in less than 24 hours are considered as belonging to the same ICU stay
and the upper bound, 72 hours, is based on medical advice obtained in previous works [11].
Data regarding patient demographics, gender and age, and measurements from the variables pre-
sented in Table 3.1 were then collected from this initial set of patients. The choice of these variables,
13
except creatinine and troponin, is based on [11], where their high predictive power was shown. The
choice of creatinine and troponin as variables of interest resulted from medical suggestions, as they are
important indicators of ICU patients’ health.
Table 3.1: List of initially considered variables from the MIMIC II database and corresponding totalnumber of measurements.
Variable ID Variable Name Total Number of Measurements
211 Heart Rate 1,407,781
677 Temperature 305,230
828 Platelets 75,213
456 Non-Invasive Arterial Blood Pressure (NBP) Mean 616428
646 Peripheral Capillary Oxygen Saturation (SpO2) 1,374,595
818 Lactic Acid (0.5-2.0) 25,799
791 Creatinine (0-1.3) 76,393
851 Troponin 815
Once obtained the initial dataset, a statistical analysis was performed with the aim of better charac-
terizing each set of patients, readmitted and not readmitted, as presented in the following sections.
3.2 Age and Gender
In the present section, the distributions of each set of patients according to their age and gender are
presented. Figure 3.1 presents the distributions of each set of patients, readmitted and not readmitted,
according to their age.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.1: Age distribution.
In terms of age distribution, there is a slight increase in the number of readmitted patients aged
between 70 and 90 years old and an even slighter increase in the number of not readmitted patients
between 50 and 70 years old.
The distributions of each set of patients, now considering both their age and gender, is again pre-
sented in Figure 3.2.
14
(a) Not readmitted patients (b) Readmitted patients
Figure 3.2: Age and gender distributions.
Regarding the gender distribution, there is mostly a predominance in the number of male patients but
can not be spotted any obvious differences between the readmitted and not readmitted patients cases,
apart from an increase in the number of readmitted female patients aged between 70 and 85 years old.
3.3 ICU Length of Stay
The total amount of time a patient is hospitalized in the ICU can be an indicator of the severity of his
medical condition. Considering this, Figure 3.3 shows the ICU lengths of stay distributions, in hours, for
each set of patients.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.3: ICU length of stay.
For both cases, a higher incidence of patients hospitalized between 24 (the minimum imposed) and
60 hours is observed, lacking the existence of a clear distinction between readmitted and not read-
mitted patients. Additionally, a cyclic behaviour of peaks can be spotted in both cases, with about 24
hours of interval, that most likely corresponds to the daily procedure of rounds where more patients are
discharged.
3.4 Number of Measurements
The number of times a given variable is measured can reflect a patient’s health situation. In this
section, Figures 3.4 to 3.11 present the distributions of number of measurements per patient for each of
15
the variables listed in Table 3.1, for readmitted and not readmitted patients.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.4: Heart rate measurements.
In terms of heart rate measurements, in Figure 3.4 it is observable a comparable distribution for
readmitted and not readmitted patients. However, it can be noted that there are more not readmitted
than readmitted patients having a number of measurements inferior to one hundred and that the opposite
happens for superior numbers of measurements, although in a less obvious way.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.5: Temperature measurements.
Regarding temperature measurements, Figure 3.5 shows that the number of patients having null or a
low amount of measurements is considerably higher for not readmitted patients than it is for readmitted
ones. Additionally, it is depicted that most of the patients with high numbers of measurements belong to
the readmitted class.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.6: Platelets measurements.
In terms of platelets measurements, a clear reduction in the total amount of measurements is pre-
16
sented in Figure 3.6. Furthermore, it can be noted again, a slightly higher incidence of patients having
null or a low amount of measurements in not readmitted patients.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.7: NBP mean measurements.
Figure 3.7 shows a substantially higher amount of not readmitted patients having less than fifty NBP
mean measurements and a specially higher number of not readmitted patients where no measurement
was made.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.8: SpO2 measurements.
Regarding the SpO2 numbers of measurements, Figure 3.8 shows a rather similar distribution for
both sets of patients. However, it can be noted a slightly higher amount of not readmitted patients
having a reduced number of SpO2 measurements.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.9: Lactic acid measurements.
In terms of lactic acid measurements, is shown in Figure 3.9 how the number of measurements for
this variable is substantially low when compared to the previous cases. However, although small, it can
17
be noted a higher incidence of not readmitted patients with null measurements.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.10: Creatinine measurements.
Regarding the creatinine measurements, Figure 3.10 shows, once again, a slightly higher incidence
of not readmitted patients having a reduced number of measurements.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.11: Troponin measurements.
Finally, for the case of troponin, presented in Figure 3.11, it can be noted how most patients do not
present any measurement, independently of their group. In fact, the lack of measurements from this
variable is so high that the variable needed to be excluded for the construction of the final dataset.
Overall, it can be concluded that most of the selected variables present differences, even if small, in
their behaviour for readmitted and not readmitted patients.
3.5 Elapsed time between admission and first measurement
The elapsed time between a patient’s admission and the first time a variable is measured can be a
strong indicator of the condition in which a patient is admitted to the ICU. In this section, Figures 3.12
to 3.19 present the distributions of the elapsed times between admission and the first measurement of
each of the variables listed in Table 3.1 for each group of patients.
18
(a) Not readmitted patients (b) Readmitted patients
Figure 3.12: Elapsed time between admissionand first heart rate measurement.
In terms of heart rate, it can be seen in Figure 3.12 how there is a considerably higher amount of
readmitted patients having the first measurement made less than an hour after being admitted to the
ICU.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.13: Elapsed time between admissionand first Temperature measurement.
Regarding the time of the first measurement of temperature, it is depicted in Figure 3.13 how both
sets of patients seem to have similar distributions. Nevertheless, it can be seen that a slightly bigger
amount of readmitted patients have their temperature measured in the first hour of their admission.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.14: Elapsed time between admissionand first platelets measurement.
Figure 3.14 shows how the distributions of time until the first measurement of platelets is rather
similar for both sets of patients. Slight differences do exist, but the percentage of patients considered is
too low for these differences to be significant.
19
(a) Not readmitted patients (b) Readmitted patients
Figure 3.15: Elapsed time between admissionand first NBP mean measurement.
In terms of time elapsed until the first measurement of NBP mean, Figure 3.15 shows how there is
a higher amount of readmitted patients with this measurement taking place in the initial hours of their
admission.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.16: Elapsed time between admissionand first SpO2 measurement.
Also in terms of the time elapsed until the first measurement of SpO2, it is depicted in Figure 3.16 a
higher incidence of readmitted patients with the first measurement taking place in the first hour of their
admission.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.17: Elapsed time between admissionand first lactic acid (0.5-2.0) measurement.
Regarding the elapsed time until the first lactic acid (0.5-2.0) measurement, in Figure 3.17 is shown
how the distribution is rather similar for both readmitted and not readmitted patients.
20
A-PDF Page Crop DEMO: Purchase from www.A-PDF.com to remove the watermark
(a) Not readmitted patients (b) Readmitted patients
Figure 3.18: Elapsed time between admissionand first creatinine (0-1.3) measurement.
For the case of creatinine (0-1.3), presented in Figure 3.18, as in the previous case there is no obvi-
ous distinction between the time elapsed until the first measurement for readmitted and not readmitted
patients.
(a) Not readmitted patients (b) Readmitted patients
Figure 3.19: Elapsed time between admissionand first troponin measurement.
At last, for the case of troponin case, the number of measurements is so low that it is hard to obtain
a good characterization of its distribution.
Although some differences have been spotted in the elapsed times until the first measurements of the
several variables considered, it can be concluded that the use of the analysis conducted in this Chapter
alone is insufficient to fully characterize and distinguish the two categories of patients. Therefore, it
can be understood how the problem of successfully predicting patients’ readmissions is complex and
extensive, requiring more complex approaches.
21
22
Chapter 4
Readmissions Data Processing
This Chapter presents an overview of the steps taken to obtain the final datasets used to implement
the data mining phase of the KDD procedure.
Initially, the steps of data acquisition and data preprocessing are covered, followed by a description
of the assumptions made and the procedure followed to obtain the totality of samples and respective
outputs. The two approaches considered to derive the final datasets are described, the use of prime
and extracted features, and the data normalization step is explained.
4.1 Data Acquisiton
As discussed in Section 3.1, a dataset containing information regarding the measurements of the
variables presented in Table 3.1 was initially extracted from the MIMIC II database. This dataset con-
cerned patients that survived within one year of discharge and that were ICU inpatients for a minimum of
24 hours. Patients who returned to any ICU of the same medical center between 24 and 72 hours were
classified as readmitted patients and the ones that did not return to any ICU, classified as not readmitted
patients, resulting in a set composed by 12,080 not readmitted patients and 775 readmitted patients.
Having been shown in the previous analysis the lack of available measurements of the variable
Troponin, data concerning it has not been included in the following steps. Hence, the set of initial patients
was restricted to the ones containing a minimum of two measurements of the variables presented in
Table 4.1, resulting in a set comprising 2,645 not readmitted patients and 241 readmitted patients.
23
Table 4.1: Final variables considered from the MIMIC II database.
Variable ID Variable Name
211 Heart Rate
677 Temperature
828 Platelets
456 NBP Mean
646 SpO2
818 Lactic Acid (0.5-2.0)
791 Creatinine (0-1.3)
From this set of patients were then extracted the patients’ gender, age, height, weight, time of admis-
sion in the ICU and time and value of each measurement of the variables listed in Table 4.1, as well as
an indication of the patient’s class, 0 for not readmitted patients and 1 for readmitted ones.
4.2 Data Preprocessing
In large databases, as the present one, outliers can be rather common, being necessary to handle
them so that the remaining data points do not become corrupted [2]. With this purpose, a method in
which values outside an interval of valid physiological ranges are eliminated was applied to the present
dataset. The maximum and minimum values considered for the physiological ranges of each variable
can be found in Table 4.2 [39].
Table 4.2: Physiological ranges of the variables considered from the MIMIC II database.
Variable ID Variable Name Units Minimum Maximum
211 Heart Rate beats/min 0 250
677 Temperature oC 25 42
828 Platelets cells ×103/µL 3 1,000
456 NBP Mean mmHg 10 187
646 SpO2 % 60 100
818 Lactic Acid (0.5-2.0) mg/dL 0 10
791 Creatinine (0-1.3) mg/dL 0.1 9
Additionally, some preprocessing was required regarding the patients’ age, given that in the MIMIC II
database patients older than 95 years have had their age register modified to exaggerated high values,
so that their identity remained protected. Consequently and to prevent the remaining values from being
influenced by such exaggerated records, these values have been once again modified to the maximum
of 95 years old.
Once the preprocessing was complete, the dataset was again restricted to patients with at least two
measurements of each of the variables listed in Table 4.1 and with a valid record of gender, age, height
and weight, resulting in a final dataset comprising 1499 not readmitted patients and 105 readmitted
24
patients, corresponding to 7% of the total of not readmitted patients.
A scheme describing the data acquisition and data preprocessing steps is presented in Figure 4.1.
Figure 4.1: Scheme of the data acquisition and data preprocessing steps.
4.3 Output Definition
One of the main goals of this thesis is to develop models capable of predicting patients’ readmissions
at any stage of their hospitalization in an attempt to mimic the real-world situation in which clinicians do
not know a priori when is the discharge day and have to evaluate continuously their patients until the
decision to discharge is made. To achieve such goal, it is first necessary to obtain samples representing
the different stages of hospitalization and to assign a corresponding output to each one.
Taking this into account, a sample time of 24 hours was chosen and each patient was evaluated
once a day along his stay in the ICU and at the moment of discharge. As a patient’s health condition
days before discharge is worse than that at the moment he or she leaves the ICU, all samples obtained
more than 24 hours before discharge were assigned to the readmitted patients’ class. This decision was
made considering that such instants represent moments in time where a patient is not fully treated and
would have to be readmitted in case he or she was discharged. Figure 4.2 presents the evolutions of
25
the outputs considered for samples belonging to not readmitted and readmitted patients.
discharge
1
0
Output
Days discharge
1
0
Output
Days
24h24h
(a) Not readmitted patients
discharge
1
0
Output
Days discharge
1
0
Output
Days
24h24h
(b) Readmitted patients
Figure 4.2: Output evolution.
After the sampling process, a final dataset was obtained comprising 1,144 samples from originally
readmitted patients and 10,911 samples from originally not readmitted patients, distributed by the two
classes as is shown in Figure 4.3.
105 readmitted
patients
1,144 samples classified as
readmitted patients
8,007 samples classified as
readmitted patients
2,904 samples classified as not
readmitted patients
1,499 not readmitted
patients
evaluated every 24 hours and at discharge
evaluated every 24 hours and at discharge
Figure 4.3: Scheme of samples distribution.
4.4 Prime Features Dataset
In an initial approach, a prime features dataset was constructed where the current value of each of
the physiological time series variables listed in Table 4.1 was considered for each sample.
Additionally, the number of days since admission in the ICU, tadm, the age, the gender and the body
mass index (BMI) of the belonging patient were added to each sample, resorting to Equation 4.1 to obtain
the BMI. The BMI is a measure of relative size based on the weight, w, and height, h, measurements
of an individual that can provide a better indication of a patient’s constitution than the use of either
measurement alone, having been associated in previous studies to the mortality of adult patients in
ICUs [40].
BMI =w
h2[kg/m2] (4.1)
In Table A.1 of Appendix A the set of 11 features considered for the prime features dataset is pre-
26
sented. This dataset was developed with the aim of being used as a comparison basis to other ap-
proaches.
4.5 Extracted Features Dataset
Feature extraction is a powerful technique of feature transformation, used to obtain a more relevant
representation of a given dataset [28].
With the aim of making further use of the information contained in the time series of physiological
variables, a feature extraction approach was taken. A new set of features was obtained based on medical
advice in an attempt to reproduce the information considered by clinicians when deciding on discharge.
In order to better retain the information contained in the original time series, the following features
were extracted from the series of records of each variable for each sample:
- Value of the last recorded measurement, vl;
- Value of the first recorded measurement, vf ;
- Maximum value recorded until the evaluation point, vmax;
- Time elapsed between admission and maximum value recorded until the evaluation point, tmax;
- Minimum value recorded until the evaluation point, vmin;
- Time elapsed between admission and minimum value recorded until the evaluation point, tmin;
- Mean of all values recorded until the evaluation point, vm;
Additionally to these features, the number of days since admission in the ICU, tadm, the age, the
gender and the BMI were added to each sample, resulting in a total of 53 features per sample, listed in
detail in Table A.5 of Appendix A.
A scheme comparing the features considered from the time series variables for the prime and ex-
tracted features datasets is presented in Figure 4.4.
Evaluation
point
Evaluation
point
𝑣𝑚
𝑣𝑚𝑎𝑥
𝑣𝑚𝑖𝑛𝑡𝑚𝑖𝑛
𝑡𝑚𝑎𝑥
𝑡𝑎𝑑𝑚
𝑣𝑙𝑣𝑓
𝑡𝑎𝑑𝑚
𝑣𝑙
Value Value
Time Time
(a) Prime features
Evaluation
point
Evaluation
point
𝑣𝑚
𝑣𝑚𝑎𝑥
𝑣𝑚𝑖𝑛𝑡𝑚𝑖𝑛
𝑡𝑚𝑎𝑥
𝑡𝑎𝑑𝑚
𝑣𝑙𝑣𝑓
𝑡𝑎𝑑𝑚
𝑣𝑙
Value Value
Time Time
(b) Extracted features
Figure 4.4: Features acquired from the time series variables.
27
4.6 Data Normalization
After the datasets are defined, the normalization of the data is required in order to have all data
values placed within a similar range.
To do so, a commonly used method in engineering and biomedical applications was chosen, the
minimum-maximum normalization [34, 41].
In this method, firstly, the minimum and maximum values of each feature f are computed, repre-
sented by xfminand xfmax
, respectively. Afterwards, for each sample i and feature f , the new normalized
value, xifnorm, is obtained from the prime value, xif , through Equation 4.2.
xifnorm=
xif − xfmin
xfmax − xfmin
(4.2)
28
Chapter 5
Modeling
In this Chapter, the theoretical background of the models developed is presented. Initially, the con-
cept of fuzzy modeling is introduced and the Takagi-Sugeno (TS) fuzzy models implemented are de-
scribed. Following, the technique of clustering is explained and two clustering algorithms are presented,
the Fuzzy C-Means (FCM), one of the most widely used and the Gustafson-Kessel (GK), the one chosen
for the models developed. At last, the multimodel approach is described and the set of feature selection
and decision criteria considered in the implementation of multimodels are presented.
5.1 Fuzzy Modeling
Traditional set theory requires elements to be either part of a given set or not [42]. While some
success has been achieved based on this theory, most real-world problems cannot be solved by such
approaches where the domain is mapped exclusively into two-valued variables. It is often necessary
to introduce a certain amount of fuzziness, i.e. it may be necessary to assign elements to sets with
a degree of certainty, similarly to what is done in human reasoning, when common sense is used.
Fuzzy modeling attempts to mimic such reasoning producing what is often referred to as approximate
reasoning [42]. Resorting to fuzzy sets, elements are assigned to sets in a certain degree, indicating the
certainty of their membership. Fuzzy logic is then used to deal with such uncertainties in order to infer
new facts with a respective degree of associated uncertainty.
Static or dynamic systems that make use of fuzzy sets and fuzzy logic are referred to as fuzzy
systems [43]. They can be used in different fields and for a variety of purposes, as modeling, data
analysis, prediction or control.
The main components of a fuzzy system, as presented in Figure 5.1, are a fuzzification interface, a
knowledge base, an inference engine and a defuzzification interface [44].
The fuzzification interface, or fuzzifier, is responsible for making the conversion between the inputs
and the fuzzy system. Resorting to it, the inputs are converted into linguistic values, represented by
fuzzy sets.
The knowledge base contains the main relationships between inputs and outputs. It is composed
29
by a data base where membership functions for linguistic terms are defined and a rule base, generally
represented by if-then statements.
The inference engine is responsible for computing the fuzzy output of the system, using the informa-
tion contained in the rule base and the given input value to produce an output.
Finally, the defuzzification interface, or defuzzifier, is responsible for making the connection between
the fuzzy system and the output. It is generally used to convert a fuzzy output into a required crisp one,
as in the case of binary classifiers where the output must take one of two possible values.
Input Output
Fuzzification Inference Defuzzification
Data base
Knowledge base
Rule base
If 𝒙𝟏 is 𝑨𝒋𝟏 and
… 𝒙𝑵 is 𝑨𝒋𝑵then 𝒚𝒋 = 𝒂𝒋
𝑻𝒙 + 𝒃𝒋
Figure 5.1: Configuration of fuzzy systems configuration, adapted from [44] .
5.1.1 Takagi-Sugeno Fuzzy Models
TS fuzzy models [45] consist of fuzzy rules that translate the relationships between inputs and out-
puts using fuzzy logic. In the present work, first order TS fuzzy models [44] were used, in which each
discriminant function is composed by rules of the type:
Rj : If x1 is Aj1 and ... xN is AjN
then yj = aTj x+ bj ,
where j = 1, ..., J corresponds to the rule number, x = (x1, ..., xN ) is the input vector, N is the total
number of features, Ajn is the fuzzy set for rule Rj and the nth feature, yj is the consequent function of
rule Rj , aj is the parameter vector of rule Rj and bj a scalar offset of rule Rj . The degree of activation,
βj , for the jth rule can be computed according to Equation 5.1:
βj =
N∏n=1
µAjn(x), (5.1)
where µAjn(x) : R→ [0, 1].
The final output is then determined through the weighted average of the individual outputs obtained
by each rule, resorting to Equation 5.2.
yc =
∑Jj=1 βjyj∑Jj=1 βj
=
∑Jj=1 βj(a
Tj x+ bj)∑J
j=1 βj(5.2)
The number of rules, J , is defined by the number of groups objects are sorted into and the antecedent
fuzzy sets, Ajn, are determined using fuzzy clustering in the product space of the input and output
30
variables [44]. The consequent parameters for each rule are obtained as a weighted ordinary least-
square estimate.
Considering that this is a classification problem, a final binary output y ∈ {0, 1} is required, where
0 corresponds to not readmitted patients and 1 to readmitted ones. However, the previous is a linear
consequent that results in a continuous output, yc ∈ [0, 1], being necessary to apply a threshold, t, to
transform it. In the present work such threshold will be determined in the training phase by varying its
value from 0 to 1 and accessing the values of specificity and sensitivity obtained for each threshold.
The threshold that results in the minimum difference between the two measures will be selected, as it
represents the best compromise between true positive and negative rates. Once the threshold has been
selected, the final binary output y can be determined by:
y =
0 , yc < t
1 , yc ≥ t(5.3)
5.2 Clustering
Clustering techniques seek to divide objects into groups or clusters, according to their similarities
[9]. They belong to the unsupervised learning methods, as the division is performed without making
use of prior information about the class of each object. The goal is to divide the dataset in a way that
objects belonging to the same cluster are as similar as possible and as dissimilar as possible to objects
belonging to different clusters. Such similarity measures can be defined in different ways, according to
the purpose of the problem, but are often expressed in terms of a distance norm that can be measured
between the data vectors or between the data vectors and the centers of the clusters, the prototypes.
These prototypes, as the partitioning of the data into each cluster, are unknown a priori and are sought
out by cluster algorithms in an iterative way. Clustering algorithms are useful in situations where few prior
knowledge about the data is available, having great potential to reveal underlying structures in it. They
work in an attempt to reconstruct these unknown structures so to obtain the best division of the data into
clusters representing different categories. They can be used for classification and pattern recognition
purposes, but also to help reduce the complexity of modeling and optimization problems [9].
There are several types of clustering algorithms that can be roughly distinguished into two categories
according to the way the data is partitioned: hard and fuzzy clustering [9]. Hard clustering makes use
of the classical set theory, requiring objects to be either part of a cluster or not. This implies that the
data is partitioned into mutually exclusive subsets. On the other hand, fuzzy or soft clustering enables
the division of the data into several clusters simultaneously, with different degrees of membership. Their
values range from 0 to 1, indicating the degree of memberships to each cluster. In many cases this
approach can be more suitable than the previous, as objects on the boundaries between several classes
are not required to belong exclusively to one of the classes.
31
5.2.1 Fuzzy C-Means
Fuzzy clustering algorithms based on objective functions have been widely used for different pur-
poses, as pattern recognition, data mining, image processing and fuzzy modeling [46]. Such methods
partition the datasets into overlapping groups in order to obtain clusters describing its underlying struc-
tures. When dealing with these methods, a number of aspects should be considered, namely the shape
and volume of the clusters, the initialization of the algorithm, the distribution of the data patterns and the
number of clusters.
There is a wide variety of clustering algorithms. One of the most popular and commonly used is the
FCM algorithm, based on the minimization of an objective function [47], as it will be discussed further
on.
In the FCM algorithm, a predefined number of clusters, nc , is computed, where each cluster is
characterized by a prototype, vi, defined by:
vi = [v1, ..., vq], (5.4)
where i = 1, 2, ..., nc and q represents the total number of features considered.
Each data sample, xj , is assigned with a normalized membership degree, µij , to a cluster i, being
the membership degrees contained in a fuzzy partition matrix, U :
U =
µ11 · · · µ1Ns
.... . .
...
µnc1 · · · µncNs
, (5.5)
where Ns is the total number of samples in the dataset.
The prototypes, vi, represent the centers of the clusters and can be computed according to:
vi =
∑Ns
j=1 µmijxj∑Ns
j=1 µmij
, (5.6)
where m ∈ [0,∞[ is the weighting exponent that determines the degree of fuzziness of clusters. High
values of m mean soft boundaries between clusters, while lower values stand for harder boundaries.
In the present work, each sample is assigned to each cluster with a certain degree of membership.
This degree is proportional to the distance between the sample and the cluster prototype, which in a
general way can be computed as:
d2ij(xj , vi) = ‖xj − vi‖2 = (xj − vi)TAi(xj − vi), (5.7)
where Ai is a positive definite symmetric matrix, usually equal to the identity matrix in the FCM algorithm.
Once the distances are computed, the referred membership degrees can be obtained through:
µij =1∑nc
k=1(dijdkj
)2
m−1
(5.8)
32
It should be noted that µij ∈ [0, 1], where zero implies that the sample j does not belong at all to
cluster i, in opposition to one that implies that a sample j completely belongs to cluster i. Additionally, it
must be considered that the sum of all membership degrees of any sample to all clusters must be equal
to one, according to:
nc∑i=1
µij = 1, ∀j, (5.9)
and that the total sample memberships to any cluster must be bigger than zero and smaller than one
according to:
0 <
Ns∑j=1
µij < Ns, ∀i (5.10)
The problem of assigning a dataset, X, into nc clusters is considered optimal when the previous
mentioned FCM objective function is minimized. This objective function is given by:
Jm(X,U, V ) =
nc∑i=1
Ns∑j=1
µmijd2ij(xj , vi), (5.11)
where V is a matrix containing all cluster prototypes vi.
Such minimization represents a nonlinear optimization problem that can be solved using Picard iter-
ation through the first-order conditions for stationary points of the objective function, Jm.
5.2.2 Gustafson-Kessel
As mentioned previously, in the FCM algorithm, the matrix Ai used to compute the distance in Equa-
tion 5.7 is the identity matrix. This causes the distance measure used to be reduced to the Euclidean
distance, making it suitable for clusters with spherical shapes [48].
However, the clusters obtained for a given dataset can have different geometrical shapes and orien-
tations. Considering these factors, the GK algorithm [49] can be a good alternative, since it resorts to
the Mahalonibis distance [46], where Ai is represented by Equation 5.12:
Ai = P−1i , (5.12)
Pi is the covariance matrix of cluster i and one additional volume constrain, |Ai| = ρi, must be added.
The objective function is then identical to the one used in the FCM algorithm, Equation 5.11, as
the update equations for the cluster centers, Equation 5.6, and the update equations for the member-
ship degrees, Equation 5.8. However, the alteration introduced in the distance measure improves the
adaptability to different cluster shapes, making it the fuzzy clustering algorithm chosen.
33
5.3 Multimodel Approach
The problem of modeling is usually associated with selecting the best model and being conditioned to
produce inferences from it [50]. However, a different approach can be taken by selecting more than one
model and producing inferences based on the entire set of models. Such approach is named multimodel,
or multiple model approach [51].
In recent decades, many studies were developed considering this approach. In a variety of areas, as
weather predictions [52], predicting the necessity of vasopressors in sepsis patients [24] or predicting
patients’ readmissions into ICUs [25], better results were achieved using multimodels than the traditional
single model.
In this work, a multimodel with a posteriori decision will be used to predict patients’ readmissions,
following the procedure presented in [24].
A posteriori decision means that the aggregation into the final output is carried out only after testing
each model independently. In a general way, it can be said that the models are tested separately, each
predicting an output. The set of predicted outputs is then combined into one single output, according to
a preselected decision criterion.
The present multimodel is composed by two independent models, one developed with the aim of
maximizing the sensitivity, i.e. the amount of readmitted patients correctly classified and the other with
the aim of maximizing the specificity, the amount of not readmitted patients correctly classified.
Figure 5.2 presents an overall scheme of the multimodel implemented to perform the patients’ clas-
sification.
Figure 5.2: Multimodel configuration with a posteriori decision, adapted from [25, 26] .
5.3.1 Model Selection
Selecting models that maximize sensitivity and specificity can be a complicated task, since many
different criteria can be used and, to the best of the author’s knowledge, there is none considered as
the most correct one. Hence, with the aim of exploring different possibilities, the following three sets of
feature selection performance criteria are used to develop the models featuring the multimodel:
1. Areas under the sensitivity and specificity curves
2. Sensitivity and specificity at the intersection threshold
3. Sensitivity and specificity close to the intersection threshold
34
In the first set, the specificity and sensitivity values for the test samples are computed for each
threshold α, sα and spα, respectively, being the threshold varied between 0 and 1 and the overall sum
of sensitivity and specificity values considered. In the model highlighting sensitivity, a bigger weighting
factor, wφ, is given to the sum of the sensitivity values and a smaller one, wβ , to the sum of the specificity
values, being the opposite for the model highlighting specificity. In this way, more importance is given
to the measure aimed to highlight, but at the same time, a compromise is found so that the opposite
measure does not present too poor results. Hence, the performance criteria used in feature selection
for the specificity and sensitivity models, cs and csp, are calculated respectively through Equations 5.13
and 5.14, where the values found for the weighting factors wφ and wβ were, respectively, 0.8 and 0.2.
cs = wφ
1∑α=0
sα + wβ
1∑α=0
spα (5.13)
csp = wφ
1∑α=0
spα + wβ
1∑α=0
sα (5.14)
In the second set of criteria, the threshold is again varied between 0 and 1, but this time in the
training phase. The sensitivity and specificity values of the training samples are calculated for each
threshold, as well as the difference between the two. The threshold, α, corresponding to the minimum
difference is chosen and kept for the test phase. In this set of criteria, solely the values of sensitivity
and specificity corresponding to that threshold are computed and, once again, more weight, w1, is given
to the sensitivity value in the model highlighting sensitivity and the opposite for the model highlighting
specificity. In this way, it is aimed to obtain the models that result in the best values of sensitivity and
specificity at the intersection point of both measures. The performance criteria used to perform feature
selection for the specificity and sensitivity models, are calculated respectively through Equations 5.15
and 5.16.
cs = wφsα + wβspα (5.15)
csp = wφspα + wβsα (5.16)
Finally, in the third set of criteria considered, the threshold is found in the same way as in the previous
one. However, when accessing the values of sensitivity and specificity in the test phase, a slight variation
of the threshold is chosen. For the model highlighting sensitivity the threshold α becomes the one
selected less 0.01 and in the one highlighting specificity, 0.01 is added to the initial threshold instead.
Such criteria were chosen with the aim of intensifying the importance given to each measure, since
values of sensitivity increase as the threshold decreases to zero and the opposite for the values of
specificity. The overall expressions to calculate the sensitivity and the specificity criteria are the same
as those used in Equations 5.15 and 5.16, respectively.
At last, it should be added that the weighting factors wφ and wβ chosen for the last two criteria were,
respectively, 0.7 and 0.3. These values differ from the first ones as the sole values of sensitivity and
specificity obtained for the selected thresholds were much more uneven than the ones obtained through
35
the overall sums, thus being necessary to adjust the corresponding weighting factors, obtained in an
empirical way throughout the work.
5.3.2 Decision Criterion
To obtain the final output of a multimodel is necessary to apply a decision criterion to combine, or
select, the outputs obtained by each of the integrating models. In the multimodel implemented, again
with the aim of testing different possibilities, three distinct decision criteria are considered to produce the
patients’ classification.
The first one consists of averaging the outputs of each model. That is, for an output y1 of the
sensitivity model and an output y2 of the specificity model, the output yδ provided by the decision criterion
is given by Equation 5.17.
yδ =y1 + y2
2(5.17)
The second criterion consists of choosing the predicted outcome with the lower uncertainty asso-
ciated, as it is presented in [25]. For each model i a threshold αi is found, according to the feature
selection criteria used. For the case of the first set of criteria where the overall sums of sensitivity and
specificity are considered, the threshold chosen is the same as the one used in the second set of criteria
where the intersection of sensitivity and specificity is sought. The differences between the outputs yi
and the respective threshold are then computed. The output corresponding to the maximum difference
corresponds to a predicted outcome closer to its predicted class and, consequently, with a lower degree
of uncertainty associated. In this way, the decision is performed according to:.
yδ = maxi|αi − yi| (5.18)
The third criterion consists of combining the two predicted outcomes with different weights wi, in-
versely proportional to their degree of uncertainty. In this way, more importance is given to the outcome
predicted with a lower degree of uncertainty. The weights, wi, can be computed as:
wi =
|αi − yi|αi
, yi < αi
|αi − yi|1− αi
, yi ≥ αi(5.19)
And the output yδ is given by:
yδ = w1y1 + w2y2 (5.20)
It should be noted, however, that the output yδ resulting from the decision criterion is a continuous
output yδ ∈ [0, 1], being still necessary to find a final threshold δ to transform it in the binary output
y ∈ {0, 1}. To do so, again the threshold that results in the minimum difference between the sensitivity
and specificity values of the training samples is chosen.
36
Chapter 6
Results
In this Chapter, the main results obtained for the different methods and respective approaches con-
sidered are summarized and compared.
Two distinct methods are implemented, one performing a daily evaluation where the complete set of
samples obtained in Section 4.3 is used and another carrying out a discharge day evaluation where only
samples obtained at discharge days are used in the testing sets. The daily evaluation is carried out with
the aim of developing suitable models for the prediction of patients’ readmissions at any stage of their
ICU hospitalization. The discharge day evaluation approach is in line with previous works, where patients
were evaluated solely at their discharge day. In particular, in [11], an AUC of 0.72±0.04, an accuracy of
0.71±0.03, a sensitivity of 0.68±0.02 and a specificity of 0.73±0.03 were obtained, whereas in [25] an
AUC of 0.74±0.06, an accuracy of 0.74±0.07, a sensitivity of 0.74±0.16 and a specificity of 0.74±0.09
were obtained.
For each evaluation, results are presented in the following way: initially, the prime features dataset
is used in a single model configuration. Then, the same model configuration is kept but the extracted
features dataset is used instead, so that conclusions can be drawn regarding the use of prime and
extracted features. At last, the extracted features are used in a multimodel configuration, so that the
influence of this configuration can be studied and compared with the one resulting from the use of a
single model with the same features.
In the development all the tested approaches, the same methodology was followed in order to obtain
comparable results. For the same reason, the type of models developed was also kept the same, having
been used TS fuzzy models where the number of rules and the antecedent fuzzy sets were determined
using the GK fuzzy clustering in the product space of the input and output variables.
In terms of the methodology followed, initially the readmitted and not readmitted patients and their
respective samples were divided randomly among two subsets, one for feature selection (FS) and the
other to perform model assessment (MA). This division was carried out in a way that approximately the
same number of readmitted and not readmitted patients, along with their samples, was assigned to each
subset .
The FS subset was firstly used to decide on the parameters of the model to be developed, the number
37
of clusters, nc, and the degree of fuzziness, m, by creating models and using the respective AUC as
performance criterion. Once the model parameters selected, this subset was used to perform feature
selection with the SFS technique, resorting to the AUC as criterion in the single model case and the
criteria presented in Section 5.3.1 in the multimodel one.
Having selected the model parameters and best subsets of features, the MA subset was used to
develop models and evaluate their performance in terms of AUC, accuracy, sensitivity and specificity. In
this way , different data from the one where the parameters and features were selected was used, so
that the results obtained are independent from the previous phase and comparable with each other.
With the aim of reducing the variability of the results and to assure their independence from the
training and test groups selected, the K-fold cross-validation methodology was used in all steps of the
model development and assessment. With it , the patients and their respective samples are separated
into K folds, in a way that an approximate equal proportion of readmitted and not readmitted patients is
kept in each fold. Once the data is divided into the K folds, the process is repeated K times, each time
using a different fold as testing set.
6.1 Daily Evaluation
The totality of samples obtained in Section 4.3 for the different hospitalization days was used with
the purpose of developing models capable of predicting patients’ readmissions at any stage of their
hospitalization, in an attempt to mimic the real ICU situation.
6.1.1 Single Model with Prime Features
As a first approach, a single model was developed using the prime features dataset introduced in
Section 4.4.
Parameter Selection
To select the nc and m parameters, 5-fold cross-validation was used in the FS subset. The nc
parameter was varied between 2 and 10, simultaneously with the degree of fuzziness m, varied between
1.1 and 2, with a step of 0.1. Models were created, and the AUC was used as performance measure.
The complete set of results obtained can be consulted in Tables A.2 and A.3 of Appendix A, where the
best combination of parameters is presented in Table 6.1.
Table 6.1: Best combination of parameters obtained with prime features after 5-fold cross-validation.
nc m AUC
4 1.6 0.71 ± 0.02
38
Feature Selection
To select the best subset of features, the SFS method was used, together with 5-fold cross-validation.
The AUC was used as performance criterion along the SFS, and the entire process was repeated 10
times. The three subsets that presented the best results are listed in Table 6.2, being the features that
integrate each subset listed in Table A.4 of Appendix A.
Table 6.2: Best subsets of features obtained by performing SFS with 5-fold cross-validation with primefeatures.
Subset 1 Subset 2 Subset 3
Number of Features 7 7 8
AUC 0.74 ± 0.01 0.73 ± 0.01 0.73 ± 0.02
Model Assessment
For MA, the three subsets presented in Table 6.2, together with the entire initial set composed by 11
features, were used to create models and evaluate their performance using different criteria. The results
obtained for AUC, accuracy, sensitivity and specificity after performing 5 times 5-fold cross-validation are
presented in Table 6.3.
Table 6.3: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA subset.
Initial Set Subset 1 Subset 2 Subset 3
AUC 0.69 ± 0.02 0.71 ± 0.02 0.72 ± 0.02 0.72 ± 0.03
Accuracy 0.64 ± 0.02 0.66 ± 0.02 0.66 ± 0.02 0.66 ± 0.03
Sensitivity 0.64 ± 0.03 0.66 ± 0.03 0.66 ± 0.04 0.66 ± 0.04
Specificity 0.64 ± 0.04 0.66 ± 0.04 0.66 ± 0.04 0.66 ± 0.04
6.1.2 Single Model with Extracted Features
In a second approach, single models were developed with the features obtained through feature ex-
traction, explained in detail in Section 4.5.
Parameter Selection
Initially, models were created and evaluated using 5-fold cross-validation in the FS subset to select
the nc and m parameters. The two parameters were varied simultaneously, the nc between 2 and 10 and
the m between 1.1 and 2, with a step of 0.1, being used the AUC as performance measure. The best
combination of parameters achieved is presented in Table 6.4 and the results obtained for the remaining
combinations are available in Tables A.6 and A.7 of Appendix A.
39
Table 6.4: Best combination of parameters obtained with extracted features after 5-fold cross-validation.
nc m AUC
3 1.5 0.74 ± 0.02
Feature Selection
To select the best subset of features, 5-fold cross-validation was carried out, using the SFS approach.
The AUC was used as a performance criterion through the SFS and the entire process was repeated 10
times. The three subsets that presented the best results are listed in Table 6.5 and the lists of features
that compose each subset are available in Table A.8 of Appendix A.
Table 6.5: Best subsets of features obtained by performing SFS with 5-fold cross-validation withextracted features.
Subset 1 Subset 2 Subset 3
Number of Features 25 26 18
AUC 0.78 ± 0.01 0.78 ± 0.01 0.77 ± 0.02
Model Assessment
For MA, the three subsets presented in Table 6.5 were evaluated, as well as the entire initial set com-
posed by 53 features. The results obtained for AUC, accuracy, sensitivity and specificity after performing
5 times 5-fold cross-validation are presented in Table 6.6.
Table 6.6: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset.
Initial Set Subset 1 Subset 2 Subset 3
AUC 0.73 ± 0.03 0.75 ± 0.02 0.76 ± 0.02 0.76 ± 0.02Accuracy 0.65 ± 0.03 0.67 ± 0.03 0.67 ± 0.02 0.68 ± 0.02Sensitivity 0.62 ± 0.04 0.66 ± 0.04 0.65 ± 0.04 0.67 ± 0.04Specificity 0.70 ± 0.04 0.71 ± 0.04 0.72 ± 0.03 0.70 ± 0.04
6.1.3 Multimodel with Extracted Features
At last, the extracted features dataset presented in Section 4.5 was used in the implementation of
a multimodel, following the methodology and criteria presented in Section 5.3. As the initial features
considered were the same as the ones used in the single model with extracted features, the parameters
nc and m were kept the same, as presented in Table 6.4, 3 and 1.5 respectively.
40
6.1.3.1 Areas Under the Sensitivity and Specificity Curves
Feature Selection
To select the best subset of features for each model, 5-fold cross-validation was used and initially the
first set of feature selection criteria presented in Section 5.3.1 was considered. Again the SFS process
was carried out 10 times, being the best subsets obtained for the models highlighting sensitivity and
specificity presented, respectively, in Tables 6.7 and 6.8. The features that integrate each subset can be
consulted in Table A.9 of Appendix A.
Table 6.7: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the first set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 14 12 13
0.8∑1α=0 sα + 0.2
∑1α=0 spα 0.72 ± 0.01 0.71 ± 0.01 0.71 ± 0.01
Table 6.8: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the first set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 15 14 16
0.8∑1α=0 spα + 0.2
∑1α=0 sα 0.44 ± 0.01 0.44 ± 0.01 0.43 ± 0.01
Model Assessment
To conduct the MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained
for the model highlighting sensitivity, Table 6.7, and the one highlighting specificity, Table 6.8, were
combined and the three decision criteria presented in Section 5.3.2 were used. The results obtained
for the best three combinations of subsets are presented in Table 6.9 and the ones obtained for the
remaining combinations are available in Table A.12 of Appendix A.
41
Table 6.9: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the first set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.77 ± 0.02 0.76 ± 0.01 0.77 ± 0.01Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.70 ± 0.02 0.69 ± 0.02
and specificity subset 2 Sensitivity 0.69 ± 0.03 0.70 ± 0.03 0.68 ± 0.03
Specificity 0.70 ± 0.03 0.69 ± 0.03 0.71 ± 0.03
AUC 0.77 ± 0.02 0.77 ± 0.01 0.77 ± 0.01Sensitivity subset 1 Accuracy 0.70 ± 0.02 0.71 ± 0.02 0.68 ± 0.02
and specificity subset 3 Sensitivity 0.69 ± 0.03 0.71 ± 0.03 0.67 ± 0.03
Specificity 0.71 ± 0.03 0.69 ± 0.03 0.73 ± 0.03
AUC 0.77 ± 0.01 0.76 ± 0.01 0.77 ± 0.01Sensitivity subset 3 Accuracy 0.70 ± 0.01 0.71 ± 0.01 0.68 ± 0.01
and specificity subset 3 Sensitivity 0.70 ± 0.02 0.72 ± 0.02 0.67 ± 0.02
Specificity 0.70 ± 0.03 0.68 ± 0.03 0.73 ± 0.03
6.1.3.2 Sensitivity and Specificity at the Intersection Threshold
Feature Selection
In a different approach, the second set of feature selection presented in Section 5.3.1 was used
with 5-fold cross-validation to select the best subsets of features for each model. SFS was performed
10 times and the best three subsets obtained for each model were kept. Table 6.10 presents the best
subsets of features obtained for the model highlighting sensitivity and Table 6.11 the best subsets of
features obtained for the model highlighting specificity. Information about the features that compose
each subset can be found in Table A.10 of Appendix A.
Table 6.10: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the second set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 21 10 12
0.7 sα + 0.3 spα 0.70 ± 0.02 0.69 ± 0.02 0.68 ± 0.03
Table 6.11: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the second set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 13 15 19
0.7 spα + 0.3 sα 0.69 ± 0.01 0.69 ± 0.02 0.69 ± 0.02
42
Model Assessment
To conduct the MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained
for the models highlighting sensitivity and specificity, Tables 6.10 and 6.11, were combined and the
three decision criteria presented in Section 5.3.2 were used. The results obtained for the best three
combinations of subsets are presented in Table 6.12. Additionally, the results obtained for the remaining
combinations are available in Table A.13 of Appendix A.
Table 6.12: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the second set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.76 ± 0.01 0.76 ± 0.01 0.76 ± 0.01Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.71 ± 0.02 0.67 ± 0.02
and specificity subset 1 Sensitivity 0.69 ± 0.03 0.72 ± 0.03 0.66 ± 0.03
Specificity 0.69 ± 0.03 0.67 ± 0.03 0.72 ± 0.03
AUC 0.76 ± 0.01 0.76 ± 0.01 0.76 ± 0.01Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.71 ± 0.02 0.67 ± 0.02
and specificity subset 3 Sensitivity 0.68 ± 0.03 0.71 ± 0.03 0.65 ± 0.04
Specificity 0.71 ± 0.03 0.68 ± 0.03 0.73 ± 0.03
AUC 0.76 ± 0.02 0.76 ± 0.02 0.76 ± 0.02Sensitivity subset 3 Accuracy 0.69 ± 0.02 0.71 ± 0.02 0.68 ± 0.03
and specificity subset 3 Sensitivity 0.69 ± 0.03 0.72 ± 0.03 0.65 ± 0.04
Specificity 0.70 ± 0.03 0.67 ± 0.03 0.73 ± 0.03
6.1.3.3 Sensitivity and Specificity Close to the Intersection Threshold
Feature Selection
At last, the third set of feature selection criteria presented in Section 5.3.1 was used, being repeated
10 times the SFS procedure with 5-fold cross-validation. Table 6.13 presents the three best subsets of
features obtained for the model highlighting sensitivity and Table 6.14 the best three subsets of features
obtained for the model highlighting specificity. The list of the features that integrate each subset can be
found in Table A.11 of Appendix A.
Table 6.13: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the third set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 13 15 12
0.7 sα + 0.3 spα 0.69 ± 0.02 0.68 ± 0.03 0.66 ± 0.02
43
Table 6.14: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the third set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 16 16 15
0.7 spα + 0.3 sα 0.70 ± 0.02 0.70 ± 0.02 0.70 ± 0.03
Model Assessment
To conduct the model assessment, 5-fold cross-validation was performed 5 times. The subsets of
features obtained for the model highlighting sensitivity, Table 6.13, and the one highlighting specificity,
Table 6.14, were combined and the three decision criteria presented in Section 5.3.2 were used. Ta-
ble 6.15 presents the results obtained for the best three combinations of subsets and Table A.14 of
Appendix A the results for the remaining combinations.
Table 6.15: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the third set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02Sensitivity subset 1 Accuracy 0.69 ± 0.03 0.71 ± 0.02 0.67 ± 0.03
and specificity subset 2 Sensitivity 0.69 ± 0.03 0.72 ± 0.03 0.64 ± 0.04
Specificity 0.70 ± 0.03 0.66 ± 0.03 0.74 ± 0.03
AUC 0.74 ± 0.02 0.74 ± 0.02 0.74 ± 0.02Sensitivity subset 2 Accuracy 0.69 ± 0.02 0.70 ± 0.03 0.67 ± 0.02
and specificity subset 2 Sensitivity 0.68 ± 0.04 0.71 ± 0.04 0.65 ± 0.03
Specificity 0.70 ± 0.03 0.66 ± 0.03 0.73 ± 0.03
AUC 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02Sensitivity subset 3 Accuracy 0.69 ± 0.03 0.71 ± 0.02 0.67 ± 0.02
and specificity subset 2 Sensitivity 0.68 ± 0.03 0.72 ± 0.03 0.64 ± 0.04
Specificity 0.70 ± 0.03 0.66 ± 0.03 0.74 ± 0.04
6.1.4 Results Comparison
Starting with the single model with prime features, Section 6.1.1, a slight improvement can be noted
in Table 6.3, specially in terms of AUC, resulting from the use of subsets of features selected by SFS
instead of the entire initial set. Although different subsets were tested, probably due to the close number
of selected features, the results obtained were very similar for the three subsets, being of about 0.72 for
the AUC and 0.66 for the remaining performance measures.
In Section 6.1.2, single models were developed using extracted features. Again, in Table 6.6 it can
be noted the positive influence of the use of feature selection, as the results obtained for the three
subsets of features were better than the ones obtained for the initial set. Comparing the performance of
44
these subsets to the ones obtained using prime features also some improvements can be noted, having
the greatest improvement been achieved in the measures of AUC and specificity. The values of AUC
improved from an average of 0.72 to 0.76 and of specificity improved from an average of 0.66 to 0.71.
In terms of sensitivity, the average values remained the same, about 0.66, but in terms of accuracy also
a slight improvement was achieved, from 0.66 to an average of 0.67.
In Section 6.1.3 multimodels were developed based on the extracted features, using different feature
selection and decision criteria. Overall, it can be concluded that the set of feature selection criteria that
presented the best results was the first one, using the areas under the sensitivity and specificity curves,
Table 6.9, followed by the second one, using the sensitivity and specificity values at the intersection
threshold, Table 6.12.
In terms of the decision criteria used, a similar performance was obtained for the multimodels devel-
oped with the three sets of feature selection criteria. In general, better results were achieved for the one
using the maximum distance. However, it can be seen how this criteria disfavors slightly the specificity.
The first criterion, the weighted distance, can be used in an alternative, as it achieves more balanced
results, although slightly worst than the ones from the maximum distance criterion. The average criterion
is the one that in general performed worst, favoring, however, the values of specificity.
In Table 6.16, the values of AUC obtained for the best multimodels developed for each feature selec-
tion and decision criterion are presented and in Table 6.17 the p-values resulting from each combination
of multimodels are presented.
Table 6.16: Mean and standard deviation of the AUC values obtained for the best multimodelsdeveloped.
Weighted distance Maximum distance Average
Areas under the sensitivity and the (1) (2) (3)
specificity curves - sensitivity subset 1 0.77 ± 0.02 0.77 ± 0.01 0.77 ± 0.01
and specificity subset 3
Sensitivity and specificity at the (4) (5) (6)
intersection threshold - sensitivity 0.76 ± 0.02 0.76 ± 0.02 0.76 ± 0.02
subset 3 and specificity subset 3
Sensitivity and specificity close (7) (8) (9)
to the intersection threshold - sensitivity 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02
subset 3 and specificity subset 3
45
Table 6.17: P-values between the best multimodels developed.
(1) (2) (3) (4) (5) (6) (7) (8) (9)
(1) - 0.00 0.01 0.05 0.02 0.05 0.00 0.00 0.01
(2) - - 0.00 0.04 0.05 0.13 0.01 0.00 0.02
(3) - - - 0.04 0.01 0.28 0.00 0.00 0.01
(4) - - - - 0.00 0.00 0.05 0.02 0.15
(5) - - - - - 0.01 0.08 0.03 0.25
(6) - - - - - - 0.01 0.00 0.03
(7) - - - - - - - 0.00 0.00
(8) - - - - - - - - 0.00
(9) - - - - - - - - -
It can be seen in Table 6.17 how, mostly, the different multimodels developed present p-values
smaller then 0.05 among themselves, enhancing the statistical difference between the AUC values ob-
tained for each model, with a significance level of 5%.
The best model obtained was the one using the sensitivity subset 1 and specificity subset 3, with
the areas under the sensitivity and specificity curves as feature selection criteria and the maximum
distance as decision criterion, presented in Table 6.9, identified with the number 2 in Tables 6.16 and
6.17. This model performed better than the single models with extracted features in all measurements
but specificity, having achieved 0.77 in AUC, 0.71 in accuracy, 0.71 in sensitivity and 0.69 in specificity.
In Figure 6.1, the results obtained in terms of the different performance measures considered for the
best models obtained are presented.
Figure 6.1: AUC, accuracy, sensitivity and specificity obtained with the different approachesimplemented.
At last, in Table 6.18, a comparison between the results obtained for the best model developed in the
present work and the results obtained in previous ones is made.
46
Table 6.18: Performance comparison of different models.
Multimodel with [11] [25]
extracted features
AUC 0.77 ± 0.01 0.72 ± 0.04 0.74 ± 0.06
Accuracy 0.71 ± 0.02 0.71 ± 0.03 0.74 ± 0.07
Sensitivity 0.71 ± 0.03 0.68 ± 0.02 0.74 ± 0.16
Specificity 0.69 ± 0.03 0.73 ± 0.03 0.74 ± 0.09
It can be seen how compared to the work of [11], specially better results were obtained in terms of
AUC and sensitivity. In comparison to the results obtained in [25], it is shown how only in terms of AUC
an improvement was made. However, it should be noted how the standard deviation values presented
in the remaining measures are substantially higher in the work of [25]. Furthermore, it should be noted
how in the previous works only the discharge days were considered, while the present work proposes
an approach for a daily assessment of outcomes, where the modeling strategy considers the complete
evolution of a patient’s condition, since admission to discharge, making it a more suitable and useful
approach to deal with real-world applications.
6.1.5 Model Evaluation
In order to better perceive the performance of the best model obtained, sensitivity subset 1 and speci-
ficity subset 3 of Table 6.9 with the maximum distance as decision criterion, an additional performance
assessment was carried out considering the real and predicted class outcomes of different patients. This
model obtained an AUC of 0.77, an accuracy of 0.71, a specificity of 0.71 and a specificity of 0.69. In
Table 6.19, a list of the features that comprise the sensitivity subset 1 and the specificity subset 3 can
be found.
In an attempt to study the performance of this model for different situations, three patients admitted
for a short period of time, less than 5 days, were considered, together with three patients admitted for a
medium amount of time, between 6 and 15 days, and three patients admitted for a long period of time,
that is, superior to 16 days. In each group of patients, a readmitted patient and two not readmitted ones
were included.
47
Table 6.19: Features selected for the sensitivity subset 1 and the specificity subset 3.
Sensitivity Subset 1 Specificity Subset 3
Minimum Creatinine (0 - 1.3) value Minimum Creatinine (0 - 1.3) value
Time of maximum Lactic Acid (0.5 - 2) value Time of maximum Lactic Acid (0.5 - 2) value
Time of minimum Lactic Acid (0.5 - 2) value Time of minimum Lactic Acid (0.5 - 2) value
Last NBP Mean value Last NBP Mean value
Maximum NBP Mean value Time of maximum NBP Mean value
Time of maximum NBP Mean value Time of minimum NBP Mean value
Time of minimum NBP Mean value Minimum Platelets value
Maximum Platelets value Mean Platelets recorded values
Last Temperature value Last Temperature value
Maximum Temperature value Maximum Temperature value
Time of maximum Temperature value Time of maximum Temperature value
Time of minimum Temperature value Time of minimum Temperature value
Maximum Heart Rate value Last Heart Rate value
Last SpO2 value Maximum Heart Rate value
- Last SpO2 value
- First SpO2 value
The evolutions of the real and predicted outcomes of two not readmitted patients admitted for a short
period of time are presented in Figures 6.2 and 6.3 and the same evolutions for a readmitted patient is
presented in Figure 6.4.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5
Cla
ssific
atio
n
Number of Days since Admission in ICU
Not Readmitted Patient 30062
Real Value
Predicted Value
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4
Cla
ssific
atio
n
Number of Days since Admission in ICU
Not Readmitted Patient 21871
Real Value
Predicted Value
Figure 6.2: Real and predicted outcomes ofthe not readmitted patient with ID number 30062.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5
Cla
ssific
atio
n
Number of Days since Admission in ICU
Not Readmitted Patient 30062
Real Value
Predicted Value
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4
Cla
ssific
atio
n
Number of Days since Admission in ICU
Not Readmitted Patient 21871
Real Value
Predicted Value
Figure 6.3: Real and predicted outcomes ofthe not readmitted patient with ID number 21871.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5
Cla
ssific
ation
Number of Days since Admission in ICU
Readmitted Patient 31651
Real Value
Predicted Value
Figure 6.4: Real and predicted outcomes ofthe readmitted patient with ID number 31651.
Figures 6.5 and 6.6 present the evolutions of the real and predicted outcomes of two not readmitted
48
patients admitted for a medium period of time and Figure 6.7 presents these evolutions for a readmitted
patient admitted for a medium period of time.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10 11 12 13
Cla
ssific
atio
n
Number of Days since Admission in ICU
Not Readmitted Patient 20969
Real Value
Predicted Value
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6
Cla
ssific
atio
n
Number of Days since Admission in ICU
Not Readmitted Patient 27047
Real Value
Predicted Value
Figure 6.5: Real and predicted outcomes ofthe not readmitted patient with ID number 20969.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10 11 12 13
Cla
ssific
ation
Number of Days since Admission in ICU
Not Readmitted Patient 20969
Real Value
Predicted Value
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6
Cla
ssific
ation
Number of Days since Admission in ICU
Not Readmitted Patient 27047
Real Value
Predicted Value
Figure 6.6: Real and predicted outcomes ofthe not readmitted patient with ID number 27047.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7
Cla
ssific
ation
Number of Days since Admission in ICU
Readmitted Patient 32170
Real Value
Predicted Value
Figure 6.7: Real and predicted outcomes ofthe readmitted patient with ID number 32170.
At last, Figures 6.8 and 6.9 present the evolutions of real and predicted outcomes of two not readmit-
ted patients admitted for a long period of time and Figure 6.10 presents these evolutions for a readmitted
patient admitted for a long period of time.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Cla
ssific
ation
Number of Days since Admission in ICU
Not Readmitted Patient 25862
Real Value
Predicted Value
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Cla
ssific
ation
Number of Days since Admission in ICU
Not Readmitted Patient 26159
Real Value
Predicted Value
Figure 6.8: Real and predicted outcomes ofthe not readmitted patient with ID number 25862.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Cla
ssific
ation
Number of Days since Admission in ICU
Not Readmitted Patient 25862
Real Value
Predicted Value
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Cla
ssific
ation
Number of Days since Admission in ICU
Not Readmitted Patient 26159
Real Value
Predicted Value
Figure 6.9: Real and predicted outcomes ofthe not readmitted patient with ID number 26159.
49
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Cla
ssific
ation
Number of Days since Admission in ICU
Readmitted Patient 26980
Real Value
Predicted Value
Figure 6.10: Real and predicted outcomes ofthe readmitted patient with ID number 26980.
The previous figures show how for the different lengths of stay of not readmitted patients, the values
predicted by the model tend to decrease as the discharge day approaches, even if not in a linear way.
In the case of readmitted patients, although some oscillations can be spotted, in general the values tend
to stay close to one. Some decreases can also be spotted as the discharge day approaches, but not so
clear as the ones in not readmitted patients. Therefore, the model seems to present a relatively good
overall performance in terms of indicating the right moment to discharge a patient so that he does not
end up being readmitted.
6.2 Discharge Day Evaluation
In an attempt to obtain results comparable with previous works where patients were evaluated only
at the discharge day [11, 25], the previously described methodology was repeated, this time using only
samples obtained at discharge days in the testing sets.
6.2.1 Single Model with Prime Features
Initially, a single model was developed using the prime features presented in Table A.1.
Parameter Selection
To select the nc and m parameters, 5-fold cross-validation was used in the FS subset. The nc
parameter was varied between 2 and 10, along with the degree of fuzziness m, varied between 1.1
and 2, with a step of 0.1. Models were created and the AUC was used as performance measure. The
complete set of results obtained can be consulted in Tables B.1 and B.2 of Appendix B, being the best
combination of parameters obtained presented in Table 6.20.
Table 6.20: Best combination of parameters obtained with prime features after 5-fold cross-validation.
nc m AUC
3 1.4 0.66 ± 0.06
50
Feature Selection
To select the best subset of features, the SFS method was repeated 10 times, together with 5-fold
cross-validation. The AUC was used as performance criterion, being the best three subsets of features
obtained presented in Table 6.21. The list of features that integrates each subset can be consulted in
Table B.3 of Appendix B.
Table 6.21: Best subsets of features obtained by performing SFS with 5-fold cross-validation with primefeatures.
Subset 1 Subset 2 Subset 3
Number of Features 5 3 4
AUC 0.67 ± 0.07 0.67 ± 0.09 0.67 ± 0.10
Model Assessment
To perform MA, the three subsets of features presented in Table 6.21 together with the initial set
composed by 11 features were used to create models and evaluate their performance using different
criteria. Table 6.22 presents the results obtained, in terms of AUC, accuracy, sensitivity and specificity,
after performing 5 times 5-fold cross-validation.
Table 6.22: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA subset.
Initial Set Subset 1 Subset 2 Subset 3
AUC 0.61 ± 0.01 0.67 ± 0.06 0.69 ± 0.06 0.68 ± 0.06
Accuracy 0.58 ± 0.05 0.62 ± 0.04 0.64 ± 0.03 0.64 ± 0.03Sensitivity 0.59 ± 0.02 0.61 ± 0.02 0.64 ± 0.01 0.62 ± 0.02
Specificity 0.58 ± 0.05 0.63 ± 0.05 0.63 ± 0.03 0.64 ± 0.04
6.2.2 Single Model with Extracted Features
In a second approach, single models were developed with the features obtained through feature ex-
traction, presented in Section 4.5.
Parameter Selection
Initially, models were created and evaluated using 5-fold cross-validation in the FS subset to select
the nc and m parameters. The two parameters were varied simultaneously, the nc between 2 and 10
and the m between 1.1 and 2, with a step of 0.1. The AUC was used as performance measure, being
the best combination of parameters achieved presented in Table 6.23. The results obtained for the
remaining combinations of parameters are available in Tables B.4 and B.5 of Appendix B.
51
Table 6.23: Best combination of parameters obtained with extracted features after 5-foldcross-validation.
nc m AUC
4 1.1 0.66 ± 0.06
Feature Selection
To select the best subset of features, SFS was performed 10 times with 5-fold cross-validation, using
the AUC as performance criterion. The three subsets that achieved the best results are presented in
Table 6.24 and the features that compose each subset are listed in Table B.6 of Appendix B.
Table 6.24: Best subsets of features obtained by performing SFS with 5-fold cross-validation withextracted features.
Subset 1 Subset 2 Subset 3
Number of Features 20 5 6
AUC 0.71 ± 0.08 0.70 ± 0.09 0.69 ± 0.10
Model Assessment
To conduct the MA, the three subsets of features presented in Table 6.24 together with the initial set
composed by 53 features, were evaluated in terms of AUC, accuracy, sensitivity and specificity. 5-fold
cross-validation was used, being the results obtained presented in Table 6.25.
Table 6.25: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset.
Initial Set Subset 1 Subset 2 Subset 3
AUC 0.63 ± 0.08 0.66 ± 0.06 0.73 ± 0.05 0.67 ± 0.06
Accuracy 0.60 ± 0.04 0.65 ± 0.03 0.67 ± 0.05 0.65 ± 0.03
Sensitivity 0.56 ± 0.02 0.62 ± 0.01 0.69 ± 0.09 0.63 ± 0.02
Specificity 0.60 ± 0.05 0.65 ± 0.03 0.67 ± 0.06 0.65 ± 0.01
6.2.3 Multimodel with Extracted Features
At last, the extracted features presented in Section 4.5 were used to develop multimodels, following
the methodology and criteria presented in Section 5.3. As the initial features considered were the same
as the ones used in the single model with extracted features, the parameters nc and m were kept the
same, 4 and 1.1 respectively.
52
6.2.3.1 Areas Under Sensitivity and Specificity Curves
Feature Selection
To select the best subsets of features for each model, 5-fold cross-validation was used and the
first set of feature selection criteria presented in Section 5.3.1 was considered. The SFS process was
again carried out 10 times, being the best three subsets of features obtained for the models highlighting
sensitivity and specificity presented, respectively, in Tables 6.26 and 6.27. The list of features that
integrate each subset are available in Table B.7 of Appendix B.
Table 6.26: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the first set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 18 12 13
0.8∑1α=0 sα + 0.2
∑1α=0 spα 0.69 ± 0.01 0.69 ± 0.01 0.69 ± 0.01
Table 6.27: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the first set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 21 16 18
0.8∑1α=0 spα + 0.2
∑1α=0 sα 0.42 ± 0.01 0.42 ± 0.01 0.41 ± 0.01
Model Assessment
To conduct MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained
for the models highlighting sensitivity and specificity, Tables 6.26 and 6.27, were combined and the
three decision criteria presented in Section 5.3.2 were used. The results obtained for the best three
combinations of features are presented in Table 6.28 and in Table B.10 of Appendix B can be found the
results obtained for the remaining combinations.
53
Table 6.28: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the first set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.67 ± 0.09 0.66 ± 0.09 0.67 ± 0.09Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.63 ± 0.02 0.73 ± 0.02
and specificity subset 1 Sensitivity 0.61 ± 0.12 0.64 ± 0.12 0.55 ± 0.10
Specificity 0.69 ± 0.03 0.63 ± 0.03 0.75 ± 0.03
AUC 0.66 ± 0.08 0.65 ± 0.08 0.66 ± 0.08Sensitivity subset 2 Accuracy 0.67 ± 0.03 0.62 ± 0.03 0.73 ± 0.03
and specificity subset 3 Sensitivity 0.60 ± 0.12 0.63 ± 0.12 0.55 ± 0.12
Specificity 0.68 ± 0.03 0.62 ± 0.03 0.74 ± 0.03
AUC 0.69 ± 0.08 0.69 ± 0.08 0.70 ± 0.08Sensitivity subset 3 Accuracy 0.68 ± 0.04 0.63 ± 0.04 0.73 ± 0.03
and specificity subset 2 Sensitivity 0.61 ± 0.12 0.65 ± 0.12 0.54 ± 0.14
Specificity 0.68 ± 0.04 0.63 ± 0.04 0.75 ± 0.04
6.2.3.2 Sensitivity and Specificity at the Intersection Threshold
Feature Selection
In an alternative approach, the second set of feature selection criteria presented in Section 5.3.1
was used to perform feature selection, together with 5-fold cross-validation. The SFS procedure was
performed 10 times and the best three subsets of features for each model were kept. Table 6.29 presents
the best subsets obtained for the model highlighting sensitivity and Table 6.30 the best subsets obtained
for the model highlighting specificity. Additionally, the features that compose each subset can be found
in in Table B.8 of Appendix B.
Table 6.29: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the second set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 15 20 16
0.7 sα + 0.3 spα 0.71 ± 0.15 0.71 ± 0.15 0.69 ± 0.14
Table 6.30: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the second set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 14 10 11
0.7 spα + 0.3 sα 0.67 ± 0.03 0.67 ± 0.06 0.66 ± 0.05
54
Model Assessment
For MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained for the
models highlighting sensitivity and specificity, Tables 6.29 and 6.30, were combined and the three de-
cision criteria presented in Section 5.3.2 were used. Table 6.31 presents the results obtained for the
best three combinations of subsets. In Table B.11 of Appendix B, the results obtained for the remaining
combinations are available.
Table 6.31: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the second set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.69 ± 0.05 0.69 ± 0.05 0.68 ± 0.05
Sensitivity subset 2 Accuracy 0.66 ± 0.03 0.61 ± 0.03 0.72 ± 0.03and specificity subset 2 Sensitivity 0.62 ± 0.10 0.67 ± 0.08 0.50 ± 0.11
Specificity 0.66 ± 0.03 0.61 ± 0.03 0.73 ± 0.03
AUC 0.68 ± 0.06 0.67 ± 0.06 0.68 ± 0.06Sensitivity subset 3 Accuracy 0.66 ± 0.02 0.64 ± 0.04 0.70 ± 0.03
and specificity subset 1 Sensitivity 0.62 ± 0.12 0.65 ± 0.12 0.57 ± 0.14
Specificity 0.66 ± 0.03 0.63 ± 0.03 0.70 ± 0.04
AUC 0.70 ± 0.10 0.69 ± 0.10 0.71 ± 0.09Sensitivity subset 3 Accuracy 0.65 ± 0.04 0.62 ± 0.04 0.69 ± 0.04
and specificity subset 3 Sensitivity 0.67 ± 0.12 0.71 ± 0.12 0.61 ± 0.14
Specificity 0.65 ± 0.04 0.61 ± 0.04 0.70 ± 0.04
6.2.3.3 Sensitivity and Specificity Close to the Intersection Threshold
Feature Selection
At last, the third set of feature selection criteria presented in Section 5.3.1 was used to perform
feature selection, being repeated 10 times the SFS procedure with 5-fold cross-validation. In Table 6.32
the three best subsets of features obtained for the model highlighting sensitivity are presented and in
Table 6.33 the three best subsets of features obtained for the model highlighting specificity. In Table B.9
of Appendix B the lists of features that comprise each subset are available.
Table 6.32: Best subsets of features for the model highlighting sensitivity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the third set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 13 8 10
0.7 spα + 0.3 sα 0.75 ± 0.06 0.74 ± 0.07 0.72 ± 0.05
55
Table 6.33: Best subsets of features for the model highlighting specificity, obtained by performing SFSwith 5-fold cross-validation with extracted features and the third set of feature selection criteria.
Subset 1 Subset 2 Subset 3
Number of Features 24 22 16
0.7 spα + 0.3 sα 0.69 ± 0.03 0.68 ± 0.04 0.66 ± 0.02
Model Assessment
To conduct MA, 5-fold cross-validation was performed 5 times. The subsets of features obtained
for the model highlighting sensitivity, Table 6.32, and the one highlighting specificity, Table 6.33, were
combined and the three decision criteria presented in Section 5.3.2 were used. The results obtained for
the best three combinations of subsets are presented in Table 6.34 and in Table B.12 of Appendix B the
results obtained for the remaining combinations can be found.
Table 6.34: Mean and standard deviation of the best results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the third set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.68 ± 0.08 0.68 ± 0.06 0.68 ± 0.07
Sensitivity subset 1 Accuracy 0.66 ± 0.04 0.62 ± 0.03 0.71 ± 0.03and specificity subset 3 Sensitivity 0.59 ± 0.14 0.65 ± 0.12 0.52± 0.12
Specificity 0.66 ± 0.04 0.62 ± 0.03 0.72 ± 0.04
AUC 0.67 ± 0.08 0.66 ± 0.08 0.67 ± 0.07Sensitivity subset 2 Accuracy 0.66 ± 0.03 0.62 ± 0.03 0.71 ± 0.02
and specificity subset 1 Sensitivity 0.60 ± 0.16 0.66 ± 0.14 0.53 ± 0.15
Specificity 0.67 ± 0.03 0.62 ± 0.04 0.72 ± 0.03
AUC 0.67 ± 0.06 0.67 ± 0.06 0.68 ± 0.06Sensitivity subset 3 Accuracy 0.67 ± 0.03 0.64 ± 0.03 0.71 ± 0.03
and specificity subset 2 Sensitivity 0.59 ± 0.10 0.66 ± 0.11 0.53 ± 0.11
Specificity 0.67 ± 0.03 0.64 ± 0.03 0.72 ± 0.04
6.2.4 Results Comparison
Overall, two main conclusions can be drawn upon the evaluation of the models using test samples
obtained at discharge days. First, it is clear that worse results are obtained when comparing to the use
of the full hospitalization samples, and second, that worse results are also obtained in comparison to
previous studies [25, 11].
This can be due to several reasons. One is that the full dataset was used in the training steps. This
means big amounts of information concerning different stages of hospitalization were provided as exam-
ples, making it harder for models to classify the discharge days as accurately as other models trained
only with discharge samples. Another factor can be the diminished size of the readmitted patients’
56
class, of 105 patients. These patients were first divided into the FS and MA subsets and later in 5 folds
to perform cross-validation, resulting in groups of about 10 patients per folder. Such division influences
drastically the sensitivity, as slight changes in the number of correctly classified readmitted patients re-
sult in considerable changes in sensitivity, making it difficult to achieve models with even sensitivity and
specificity performances.
In terms of the single models developed, for both the prime and extracted features approaches can
be seen in Tables 6.22 and 6.25, respectively, how the use of subsets of features obtained through
feature selection resulted in better performances than the use of the initial sets. Also, it can be noted
how the increase in performance resulting from the use of extracted features was not as notorious as
the one in the previous method, using all samples in the testing set. Nonetheless, it can be noted how
the subset 2, obtained with the extracted features, presented results considerably better than the ones
obtained through prime features, with an AUC of 0.73, accuracy of 0.67, sensitivity of 0.69 and specificity
of 0.67.
In terms of the multimodels developed, no better results were obtained, nevertheless, some interest-
ing observations can be made. In terms of the sets of feature selection criteria used, a similar perfor-
mance ranking was obtained as previously. However, in this approach it was not obtained such a clear
distinction between the first and second sets of criteria. Nevertheless, again the third set of criteria, the
use of sensitivity and specificity close to the intersection thresholds was the one that performed worse.
In terms of the decision criterion, it can be noted, once again, how the one using the average disfa-
vors strongly the sensitivity. The weighted and the maximum distances criteria presented performances
relatively comparable, but, in most cases the use of the maximum distance resulted in more balanced
results in all performance measures.
In Figure 6.11, the results obtained in terms of AUC, accuracy, sensitivity and specificity for the best
models obtained in each approach are presented, where can be noted how the greatest improvement in
performance resulted from the use of extracted features. For the multimodel approach, the best model
considered was the one with the sensitivity and specificity subsets number 3 resulting from the second
set of feature selection criteria with the weighted distance decision criterion, presented in Table 6.31.
Figure 6.11: AUC, accuracy, sensitivity and specificity obtained with the different approachesimplemented.
57
58
Chapter 7
Conclusions
The present thesis was developed with the aim of predicting patients’ readmissions in ICUs, based
on patients’ demographics information and a set of physiological variables measured along the patients’
hospitalization.
The hypothesis proposed consisted of combining a multimodel approach with features extracted
from the time series of physiological variables. It was aimed to demonstrate how the use of extracted
features can be a viable option when dealing with time series of variables and how more knowledge
can be obtained through these features than from the current value of each physiological variable alone.
Besides, it was sought to explore the advantages of using multimodels instead of single ones and to
explore the results obtained by combining feature extraction with a multimodel approach. Additionally, a
new hypothesis was proposed, the use of information from the entire patient’s hospitalization instead of
solely from the discharge days, has it had been done in previous works [11, 25].
Considering initially the entire set of samples, satisfactory results were obtained. At first, with the use
of prime features in a single model configuration, increases in the results could be noted through the
use of feature selection alone, demonstrating the power of this procedure. With the addition of extracted
features, i.e. with the addition to each sample of features considering the full hospitalization period so
far, some improvements were also achieved, both when compared to the use of the entire initial set
of extracted features, before feature selection, as to the previous use of prime features. In this way, it
was demonstrated how considering information from as far as the admission moment is advantageous
and important in predicting patients’ readmissions in ICUs, contributing with more relevant information
than the use of the current values of each variable alone. Finally, multimodels were developed, being
used different feature selection criteria to obtain two models, one highlighting sensitivity and the other
highlighting specificity and different decision criteria to transform the outputs of each model into a final
one. It was demonstrated how the best set of feature selection criteria used to obtain each model was
the sum of the weighted areas under the sensitivity and specificity curves, followed by the use of the
sensitivity and specificity values at the intersection threshold and, at last, by the use of the sensitivity
and specificity values close to that same intersection threshold. In terms of the decision criteria used,
the maximum distance was the one that presented most often best results, although a slight disfavor of
59
specificity could be noted. The weighted distance criterion also presented considerably good results,
often more even in terms of sensitivity and specificity, which can be sometimes desirable. At last, the
average criterion was the one that presented worst results, showing a slight better performance in terms
of specificity, which can also be used to one’s favor in particular situations.
The best model obtained was the one using the sum of the weighted areas under the sensitivity
and specificity curves to perform feature selection and the maximum distance as decision criterion. This
model achieved an AUC of 0.77±0.01, an accuracy of 0.71±0.02, a sensitivity of 0.71±0.03 and a speci-
ficity of 0.69±0.03. Although the accuracy, sensitivity and specificity are below those presented in [25],
the proposed approach is particularly well suited to be implemented in a systems engineering design
to provide prognostic guidance for the care of the critically ill in a daily basis. Furthermore, significant
better results were obtained in terms of AUC and sensitivity compared to the ones obtained in [11].
It can be concluded that, hypothetically, a clinician could use this model in real life situations at every
stage of a patient’s hospitalization to assist the decision of discharge, in a way that the readmission risk
is minimized.
When assessing the performance of this model in patients with varying lengths of stay, it could be
observed how for not readmitted patients the outcomes of the model tended do decrease as the dis-
charge day approached and how for readmitted patients the predicted values tended to oscillate around
one, showing a much less pronounced decline, if any, as the discharge day approached.
Additionally, in an attempt to obtain results comparable with previous works, the same procedure
was carried out, using solely discharge day samples in the testing sets. This alteration resulted in worse
results, having the best model been obtained for the use of extracted features in a single model con-
figuration where the AUC achieved a value of 0.73, the accuracy of 0.67, the sensitivity of 0.69 and
the specificity of 0.67. Such outcome can be due to the use of the entire hospitalization samples in
the training sets, contrarily to what was done in previous approaches, which could have decreased the
performance of the model as the training examples provided included more information than the specific
one associated to discharge days. Additionally, the low amount of samples from readmitted patients can
have also influenced the results, leading often to much uneven values of sensitivity and specificity and
to elevated standard deviation values of sensitivity. Such is due to the fact that the sole alteration by one
patient in the total of positive classifications changes considerably the sensitivity values. Nevertheless,
the development of multimodels, even if presenting worst performances, was useful to confirm the best
feature selection and decision criteria, as a similar ranking to the previous one was obtained in terms of
performance for both the feature selection and the decision criteria.
Overall, it can be concluded that the methods implemented in this work were successful, considering
the usefulness of predicting the risk of readmission at any stage of a patient’s hospitalization. Addition-
ally, it was demonstrated how the use of extracted features and multimodels can help improve the results
in the problem of predicting patients’ readmissions in ICUs.
60
7.1 Limitations
Through the development of this thesis some limitations were found.
For one, the attribution of the readmitted label to any sample obtained more than 24 hours before
discharge can be argued against. The doubt on the validity of this assumption can result from the fact
that no real evidence confirms the hypothesis that a patient would in fact be readmitted case he/she was
discharged any time before the real discharge day. However, this assumption was made considering that
clinicians opt not to discharge patients before time mostly due to the risks of readmission, increase of
illness and death. Therefore, considering that samples obtained before the real discharge day are similar
to samples from not readmitted patients is, to a certain point, justified. Furthermore, this assumption
was made in an attempt to obtain a model suitable for real world applications where one cannot know in
advance when the discharge day is, making it a reasonable approach to deal with this situation.
Additionally, due to the complexity and large size of the database, not all steps of the work were
repeated a sufficient number of times, or in the best way, to totally mitigate the influence of the selected
train and test samples. In the case of parameter selection, where the AUC values obtained were often
extremely close for different parameters, using the AUC measure with 5-fold cross-validation might not
have been enough to find the best combinations of parameters. Also in feature selection, where the
process was carried out solely 10 times and the best three subsets were kept, there is a lack of guarantee
that the best subsets were, indeed, found. Taking into account the presented limitations and other
research suggestions further work is presented in the next section.
7.2 Future Work
Considering the work developed in this thesis, some considerations and suggestions can be made
towards future work, since not all possible paths were explored, mostly due to the computational com-
plexity of the problem.
In terms of parameter selection, some performance measures more adequate than the AUC with
cross-validation can be used to find the appropriate number of clusters. For example, several indexes
can be computed to assess the suitability of different numbers of clusters, as the Separation [53] or the
Xie-Beni [54] indexes where the optimal number of clusters should minimize the value of the index.
Also in regarding feature selection, a more consistent method is proposed in an alternative to the
repetition of the process ten times and consequent selection of the subsets that performed better in
terms of the feature selection criterion chosen. The feature section process could be repeated a greater
amount of times, somewhere in the order of one hundred, and the subsets chosen considering the
number of times they were selected. That is because, the final value obtained for the chosen criteria can
be influenced by the training and test sets used, not necessarily corresponding to the best subsets of
features. Furthermore, the method used (SFS) is easily trapped in local optimal solutions, being unable
to guarantee the selection of the optimal set of features. Using the number of repetitions of a given
subset instead would partially solve this problem, since the larger the number of times a given set is
61
selected the greater is the certainty associated with its good performance.
One other suggestion for future work is proposed, the removal of the BMI feature. Such is recom-
mended due to the great amount of missing values of height and weight in the MIMIC II database that
led to a considerable reduction in the number of patients selected for the present study. Besides, since
the BMI feature was not always selected in the feature selection process, it is believed that its removal
would strongly benefit the results, especially in the case where the low amount of readmitted patients
conditioned the obtained results.
Finally, considering the performance of the best model obtained, further developments are proposed.
It was shown how the predicted outputs of this model tend do decrease in a more pronounced way as
the discharge day approaches for not readmitted patients and how these values tend mostly to oscillate
around one for readmitted patients. Considering this, it is believed that relevant results could be obtained
by studying in detail the predicted outputs of this model in a way to more accurately understand for what
output, or sequence of outputs, a patient should indeed be discharged. Additionally, further studies
could be developed relative to the features used in this model.
62
Bibliography
[1] Andrew McAfee and Erik Brynjolfsson. Big data: the management revolution. Harvard business
review, (90):60–6, 2012.
[2] Soumen Chakrabarti, Earl Cox, Eibe Frank, Ralf Hartmut Guting, Jiawei Han, Xia Jiang, Micheline
Kamber, Sam S Lightstone, Thomas P Nadeau, Richard E Neapolitan, et al. Data Mining: Know It
All: Know It All. Morgan Kaufmann, 2008.
[3] Dongsong Zhang and Lina Zhou. Discovering golden nuggets: data mining in financial applica-
tion. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on,
34(4):513–522, 2004.
[4] Mary K Obenshain. Application of data mining techniques to healthcare data. Infection Control,
25(08):690–695, 2004.
[5] Xiang Li, Beth Plale, Nithya Vijayakumar, Rahul Ramachandran, Sara Graves, and Helen Conover.
Real-time storm detection and weather forecast activation through data mining and events process-
ing. Earth Science Informatics, 1(2):49–57, 2008.
[6] Michael JA Berry and Gordon S Linoff. Data mining techniques: for marketing, sales, and customer
relationship management. John Wiley & Sons, 2004.
[7] Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. From data mining to knowledge
discovery in databases. AI magazine, 17(3):37, 1996.
[8] Rainu Kaushal, David Blumenthal, Eric G Poon, Ashish K Jha, Calvin Franz, Blackford Middleton,
John Glaser, Gilad Kuperman, Melissa Christino, Rushika Fernandopulle, et al. The costs of a
national health information network. Annals of Internal Medicine, 143(3):165–173, 2005.
[9] Robert Babuska. Fuzzy modeling for control. Kluwer Academic Publishers, 1998.
[10] Robert Tibshirani, Trevor Hastie, Balasubramanian Narasimhan, and Gilbert Chu. Diagnosis of
multiple cancer types by shrunken centroids of gene expression. Proceedings of the National
Academy of Sciences, 99(10):6567–6572, 2002.
[11] Andre S Fialho, Federico Cismondi, Susana M Vieira, Shane R Reti, Joao MC Sousa, and Stan N
Finkelstein. Data mining using clinical physiology at discharge to predict icu readmissions. Expert
Systems with Applications, 39(18):13158–13165, 2012.
63
[12] Donald B Chalfin, Stephen Trzeciak, Antonios Likourezos, Brigitte M Baumann, R Phillip Dellinger,
DELAY-ED study group, et al. Impact of delayed transfer of critically ill patients from the emergency
department to the intensive care unit*. Critical care medicine, 35(6):1477–1483, 2007.
[13] Malcolm Elliott, Linda Worrall-Carter, and Karen Page. Intensive care readmission: A contemporary
review of the literature. Intensive and Critical Care Nursing, 30(3):121–137, 2014.
[14] Nazima Allaudeen, Jeffrey L Schnipper, E John Orav, Robert M Wachter, and Arpana R Vidyarthi.
Inability of providers to predict unplanned readmissions. Journal of general internal medicine,
26(7):771–776, 2011.
[15] Andrew L Rosenberg and Charles Watts. Patients readmitted to icus*: a systematic review of risk
factors and outcomes. CHEST Journal, 118(2):492–502, 2000.
[16] Sujin Kim, Woojae Kim, and Rae Woong Park. A comparison of intensive care unit mortality
prediction models through the use of data mining techniques. Healthcare informatics research,
17(4):232–243, 2011.
[17] Alison J Campbell, Jonathan Alistair Cook, Gillian Adey, and Brian Cuthbertson. Predicting death
and readmission after intensive care discharge. 2008.
[18] William A Knaus, Elizabeth A Draper, Douglas P Wagner, and Jack E Zimmerman. Apache ii: a
severity of disease classification system. Critical care medicine, 13(10):818–829, 1985.
[19] Andrew L Rosenberg, Timothy P Hofer, Rodney A Hayward, Cathy Strachan, and Charles M Watts.
Who bounces back? physiologic and other predictors of intensive care unit readmission. Critical
care medicine, 29(3):511–518, 2001.
[20] Philipp GH Metnitz, Fabienne Fieux, Barbara Jordan, Thomas Lang, Rui Moreno, and Jean-Roger
Le Gall. Critically ill patients readmitted to intensive care units—lessons to learn? Intensive care
medicine, 29(2):241–248, 2003.
[21] Andre S Fialho, Uzay Kaymak, Federico Cismondi, Susana Margarida Vieira, Shane R Reti,
Joao MC Sousa, and Stan N Finkelstein. Predicting intensive care unit readmissions using prob-
abilistic fuzzy systems. In Fuzzy Systems (FUZZ), 2013 IEEE International Conference on, pages
1–7. IEEE, 2013.
[22] Henian Xia, Brian J Daley, Adam Petrie, and Xiaopeng Zhao. A neural network model for mortality
prediction in icu. Proc. Computing In Cardiology, 39:261–264, 2012.
[23] Yun Chen and Hui Yang. Heterogeneous postsurgical data analytics for predictive modeling of
mortality risks in intensive care units. In Engineering in Medicine and Biology Society (EMBC),
2014 36th Annual International Conference of the IEEE, pages 4310–4314. IEEE, 2014.
[24] Jose Miguel Mourinho Rodrigues. Data mining and modeling to predict the necessity of vasopres-
sors for sepsis patients. 2013.
64
[25] Marta PB Fernandes, Claudia F Silva, Susana M Vieira, and Joao Sousa. Multimodeling for the
prediction of patient readmissions in intensive care units. In Fuzzy Systems (FUZZ-IEEE), 2014
IEEE International Conference on, pages 1837–1842. IEEE, 2014.
[26] Federico Cismondi, Abigail L Horn, Andre S Fialho, Susana M Vieira, Shane R Reti, Joao MC
Sousa, and Stan Finkelstein. Multi-stage modeling using fuzzy multi-criteria feature selection
to improve survival prediction of icu septic shock patients. Expert Systems with Applications,
39(16):12332–12339, 2012.
[27] Nura Esfandiari, Mohammad Reza Babavalian, Amir-Masoud Eftekhari Moghadam, and
Vahid Kashani Tabar. Knowledge discovery in medicine: Current issue and future trend. Expert
Systems with Applications, 41(9):4434–4463, 2014.
[28] Kari Torkkola. Feature extraction by non parametric mutual information maximization. The Journal
of Machine Learning Research, 3:1415–1438, 2003.
[29] Isabelle Guyon. Feature extraction: foundations and applications, volume 207. Springer Science &
Business Media, 2006.
[30] Pierre A Devijver and Josef Kittler. Pattern recognition: A statistical approach, volume 761.
Prentice-Hall London, 1982.
[31] Isabelle Guyon and Andre Elisseeff. An introduction to variable and feature selection. The Journal
of Machine Learning Research, 3:1157–1182, 2003.
[32] Silvia Casado Yusta. Different metaheuristic strategies to solve the feature selection problem.
Pattern Recognition Letters, 30(5):525–534, 2009.
[33] Luıs F Mendonca, Susana M Vieira, and JMC Sousa. Decision tree search methods in fuzzy
modeling and classification. International Journal of Approximate Reasoning, 44(2):106–123, 2007.
[34] Ludmila I Kuncheva. Combining pattern classifiers: methods and algorithms. John Wiley & Sons,
2004.
[35] Ewout W Steyerberg, Andrew J Vickers, Nancy R Cook, Thomas Gerds, Mithat Gonen, Nancy
Obuchowski, Michael J Pencina, and Michael W Kattan. Assessing the performance of prediction
models: a framework for some traditional and novel measures. Epidemiology (Cambridge, Mass.),
21(1):128, 2010.
[36] Thomas A Lasko, Jui G Bhagwat, Kelly H Zou, and Lucila Ohno-Machado. The use of receiver oper-
ating characteristic curves in biomedical informatics. Journal of biomedical informatics, 38(5):404–
415, 2005.
[37] Daniel J Scott, Joon Lee, Ikaro Silva, Shinhyuk Park, George B Moody, Leo A Celi, and Roger G
Mark. Accessing the public mimic-ii intensive care relational database for clinical research. BMC
medical informatics and decision making, 13(1):9, 2013.
65
[38] Derek C Angus. Grappling with intensive care unit quality-does the readmission rate tell us any-
thing? Critical care medicine, 26(11):1779–1780, 1998.
[39] Anthony S Fauci et al. Harrison’s principles of internal medicine, volume 2. McGraw-Hill Medical
New York, 2008.
[40] Maite Garrouste-Orgeas, Gilles Troche, Elie Azoulay, Antoine Caubel, Arnaud de Lassence, Chris-
tine Cheval, Laurent Montesino, Marie Thuong, Francois Vincent, Yves Cohen, et al. Body mass
index. Intensive care medicine, 30(3):437–443, 2004.
[41] Federico Cismondi, Andre S Fialho, Susana M Vieira, Joao MC Sousa, Shane R Reti, Michael D
Howell, and Stan N Finkelstein. Computational intelligence methods for processing misaligned,
unevenly sampled time series containing missing data. In Computational Intelligence and Data
Mining (CIDM), 2011 IEEE Symposium on, pages 224–231. IEEE, 2011.
[42] Andries P Engelbrecht. Computational intelligence: an introduction. John Wiley & Sons, 2007.
[43] Robert Babuska. Fuzzy systems, modeling and identification. 2001.
[44] Joao MC Sousa and Uzay Kaymak. Fuzzy decision making in modeling and control, volume 27.
2002.
[45] Tomohiro Takagi and Michio Sugeno. Fuzzy identification of systems and its applications to model-
ing and control. Systems, Man and Cybernetics, IEEE Transactions on, (1):116–132, 1985.
[46] Uzay Kaymak and Magne Setnes. Extended fuzzy clustering algorithms. ERIM Report Series
Reference No. ERS-2001-51-LIS, 2000.
[47] James C Bezdek. Pattern recognition with fuzzy objective function algorithms. Kluwer Academic
Publishers, 1981.
[48] Joseph C Dunn. A fuzzy relative of the isodata process and its use in detecting compact well-
separated clusters. 1973.
[49] D. Gustafson and W. Kessel. Fuzzy clustering with a fuzzy covariance matrix. In Proc. IEEE CDC,
pages 761–766, 1979.
[50] Kenneth P Burnham and David R Anderson. Model selection and multimodel inference: a practical
information-theoretic approach. Springer Science & Business Media, 2002.
[51] Mohammed Chadli and Pierre Borne. Multiple models approach in automation: Takagi-Sugeno
fuzzy systems. John Wiley & Sons, 2012.
[52] Claudia Tebaldi and Reto Knutti. The use of the multi-model ensemble in probabilistic climate
projections. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engi-
neering Sciences, 365(1857):2053–2075, 2007.
66
[53] Amine M Bensaid, Lawrence O Hall, James C Bezdek, Laurence P Clarke, Martin L Silbiger, John A
Arrington, and Reed F Murtagh. Validity-guided (re) clustering with applications to image segmen-
tation. Fuzzy Systems, IEEE Transactions on, 4(2):112–123, 1996.
[54] Xuanli Lisa Xie and Gerardo Beni. A validity measure for fuzzy clustering. IEEE Transactions on
pattern analysis and machine intelligence, 13(8):841–847, 1991.
67
Appendix A
Daily Evaluation
Following are presented the extended results obtained for the different approaches implemented in
the readmissions dataset. Initially, in Section A.1, the results obtained for the single models with prime
features are presented. In Section A.2, the results obtained for the single models with extracted features
are then presented and in Section A.3 the results obtained for the multimodels with extracted features
are presented.
A.1 Single Model with Prime Features
Table A.1: Prime features acquired from the MIMIC II database.
# Feature
1 Age
2 Gender
3 Number of days since admission in the ICU
4 BMI
5 Creatinine (0 - 1.3)
6 Lactic Acid (0.5 - 2)
7 NBP Mean
8 Platelets
9 Temperature
10 Heart Rate
11 SpO2
A1
Table A.2: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with prime features, varying the number of clusters between 2 and 10
and the degree of fuzziness between 1.1 and 1.5.
Degree of Fuzziness
1.1 1.2 1.3 1.4 1.5
2 0.69 ± 0.03 0.69 ± 0.03 0.69 ± 0.02 0.68 ± 0.03 0.68 ± 0.03
3 0.69 ± 0.03 0.69 ± 0.03 0.69 ± 0.02 0.69 ± 0.03 0.69 ± 0.03
4 0.68 ± 0.03 0.69 ± 0.03 0.69 ± 0.03 0.69 ± 0.02 0.70 ± 0.02
Number 5 0.69 ± 0.03 0.68 ± 0.03 0.68 ± 0.03 0.69 ± 0.02 0.70 ± 0.02
of 6 0.68 ± 0.03 0.69 ± 0.03 0.68 ± 0.03 0.69 ± 0.03 0.69 ± 0.02
Clusters 7 0.68 ± 0.03 0.69 ± 0.03 0.68 ± 0.03 0.68 ± 0.02 0.70 ± 0.02
8 0.68 ± 0.02 0.69 ± 0.03 0.68 ± 0.02 0.68 ± 0.02 0.70 ± 0.02
9 0.68 ± 0.02 0.69 ± 0.03 0.68 ± 0.02 0.68 ± 0.03 0.70 ± 0.02
10 0.68 ± 0.03 0.69 ± 0.02 0.68 ± 0.03 0.68 ± 0.03 0.68 ± 0.02
Table A.3: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with prime features, varying the number of clusters between 2 and 10
and the degree of fuzziness between 1.6 and 2.
Degree of Fuzziness
1.6 1.7 1.8 1.9 2
2 0.68 ± 0.03 0.68 ± 0.03 0.68 ± 0.03 0.68 ± 0.03 0.68 ± 0.03
3 0.70 ± 0.03 0.70 ± 0.03 0.70 ± 0.03 0.69 ± 0.03 0.69 ± 0.03
4 0.71 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.03 0.69 ± 0.03
Number 5 0.70 ± 0.02 0.70 ± 0.03 0.70 ± 0.03 0.69 ± 0.02 0.69 ± 0.03
of 6 0.70 ± 0.02 0.70 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02
Clusters 7 0.70 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02
8 0.70 ± 0.03 0.70 ± 0.03 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02
9 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02
10 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.70 ± 0.03
Table A.4: Prime features obtained for each subset, after SFS with 5-fold cross-validation.
Subset Features
1 3, 4, 5, 7, 8, 9, 11
2 3, 5, 7, 8, 9, 10, 11
3 3, 5, 6, 7, 8, 9, 10, 11
A2
A.2 Single Model with Extracted Features
Table A.5: Extracted features acquired from the MIMIC II database.
# Feature # Feature
1 Age 28 Maximum Platelets value
2 Gender 29 Time of maximum Platelets value
3 Number of days since admission in the ICU 30 Minimum Platelets value
4 BMI 31 Time of minimum Platelets value
5 Last Creatinine (0 - 1.3) value 32 Mean Platelets recordedvalues
6 First Creatinine (0 - 1.3) value 33 Last Temperature value
7 Maximum Creatinine (0 - 1.3) value 34 First Temperature value
8 Time of maximum Creatinine (0 - 1.3) value 35 Maximum Temperature value
9 Minimum Creatinine (0 - 1.3) value 36 Time of maximum Temperature value
10 Time of minimum Creatinine (0 - 1.3) value 37 Minimum Temperature value
11 Mean Creatinine (0 - 1.3) values 38 Time of minimum Temperature value
12 Last Lactic Acid (0.5 - 2) value 39 Mean Temperature recorded values
13 First Lactic Acid (0.5 - 2) value 40 Last Heart Rate value
14 Maximum Lactic Acid (0.5 - 2) value 41 Firt Heart Rate value
15 Time of maximum Lactic Acid (0.5 - 2) value 42 Maximum Heart Rate value
16 Minimum Lactic Acid (0.5 - 2) value 43 Time of maximum Heart Rate value
17 Time of minimum Lactic Acid (0.5 - 2) value 44 Minimum Heart Rate value
18 First Lactic Acid (0.5 - 2) value 45 Time of minimum Heart Rate value
19 Last NBP Mean value 46 Mean Heart Rate values
20 First NBP Mean value 47 Last SpO2 value
21 Maximum NBP Mean value 48 First SpO2 value
22 Time of maximum NBP Mean value 49 Maximum SpO2 value
23 Minimum NBP Mean value 50 Time of maximum SpO2 value
24 Time of minimum NBP Mean value 51 Minimum SpO2 value
25 Mean NBP Mean values 52 Time of minimum SpO2 value
26 Last Platelets value 53 Mean SpO2 recorded values
27 First Platelets value
A3
Table A.6: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with extracted features, varying the number of clusters between 2
and 10 and the degree of fuzziness between 1.1 and 1.5.
Degree of Fuzziness
1.1 1.2 1.3 1.4 1.5
2 0.71 ± 0.03 0.72 ± 0.02 0.73 ± 0.02 0.71 ± 0.03 0.72 ± 0.02
3 0.71 ± 0.02 0.73 ± 0.02 0.73 ± 0.03 0.73 ± 0.03 0.74 ± 0.02
4 0.71 ± 0.03 0.72 ± 0.03 0.72 ± 0.03 0.71 ± 0.02 0.72 ± 0.03
Number 5 0.71 ± 0.03 0.71 ± 0.02 0.69 ± 0.02 0.71 ± 0.02 0.72 ± 0.03
of 6 0.69 ± 0.04 0.69 ± 0.03 0.71 ± 0.02 0.71 ± 0.01 0.70 ± 0.02
Culsters 7 0.70 ± 0.03 0.70 ± 0.02 0.71 ± 0.02 0.70 ± 0.03 0.71 ± 0.01
8 0.69 ± 0.02 0.70 ± 0.02 0.69 ± 0.01 0.69 ± 0.02 0.68 ± 0.03
9 0.69 ± 0.03 0.68 ± 0.03 0.68 ± 0.02 0.71 ± 0.02 0.69 ± 0.03
10 0.67 ± 0.01 0.69 ± 0.03 0.69 ± 0.03 0.68 ± 0.03 0.70 ± 0.02
Table A.7: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with extracted features, varying the number of clusters between 2
and 10 and the degree of fuzziness between 1.6 and 2.
Degree of Fuzziness
1.6 1.7 1.8 1.9 2
2 0.71 ± 0.03 0.72 ± 0.02 0.72 ± 0.02 0.72 ± 0.02 0.72 ± 0.02
3 0.73 ± 0.02 0.72 ± 0.03 0.72 ± 0.01 0.72 ± 0.02 0.72 ± 0.02
4 0.73 ± 0.02 0.73 ± 0.03 0.72 ± 0.02 0.73 ± 0.02 0.72 ± 0.02
Number 5 0.71 ± 0.02 0.72 ± 0.02 0.71 ± 0.03 0.73 ± 0.02 0.72 ± 0.01
of 6 0.70 ± 0.02 0.70 ± 0.02 0.71 ± 0.02 0.72 ± 0.01 0.72 ± 0.01
Clusters 7 0.70 ± 0.02 0.70 ± 0.02 0.70 ± 0.02 0.71 ± 0.02 0.70 ± 0.02
8 0.70 ± 0.02 0.71 ± 0.01 0.69 ± 0.03 0.70 ± 0.03 0.69 ± 0.02
9 0.68 ± 0.03 0.70 ± 0.02 0.69 ± 0.02 0.69 ± 0.02 0.70 ± 0.02
10 0.69 ± 0.02 0.69 ± 0.03 0.69 ± 0.02 0.69 ± 0.02 0.69 ± 0.02
Table A.8: Extracted features obtained for each subset, after SFS with 5-fold cross-validation.
Subset Features
1 1, 3, 4, 6, 11, 12, 15, 17, 19, 22, 23, 24, 25, 32, 33, 35, 36, 38, 41, 42, 43, 44, 45, 47, 51
2 3, 6, 7, 11, 12, 15, 17, 19, 21, 22, 23, 24, 29, 33, 34, 35, 36, 37, 38, 40, 41, 42, 46, 47, 51, 52
3 3, 6, 11, 12, 15, 17, 19, 21, 22, 23, 24, 33, 35, 36, 38, 42, 47, 53
A4
A.3 Multimodel with Extracted Features
Table A.9: Extracted features obtained for each subset, by performing SFS with the first set of featureselection criteria, with 5-fold cross-validation.
Subset Features
Sensitivity1 9, 15, 17, 19, 21, 22, 24, 28, 33, 35, 36, 38, 42, 47
2 3, 9, 15, 17, 19, 22, 24, 33, 35, 42, 47, 51
3 13, 17, 19, 21, 24, 28, 33, 35, 36, 38, 42, 47, 51
Specificity1 9, 12, 15, 17, 19, 20, 21, 22, 24, 33, 35, 36, 38, 42, 47
2 9, 15, 17, 19, 21, 22, 24, 28, 33, 35, 36, 38, 42, 47
3 9, 15, 17, 19, 22, 24, 30, 32, 33, 35, 36, 38, 40, 42, 47, 48
Table A.10: Extracted features obtained for each subset, by performing SFS with the second set offeature selection criteria, with 5-fold cross-validation.
Subset Features
Sensitivity1 5, 8, 10, 11, 12, 15, 17, 19, 21, 22, 23, 29, 31, 33, 35, 38, 44, 47, 48, 49, 53
2 5, 15, 17, 19, 33, 35, 44, 47, 49, 50
3 11, 12, 17, 19, 28, 30, 33, 36, 42, 47, 49, 51
Specificity1 3, 12, 14, 15, 17, 20, 22, 31, 33, 39, 40, 47, 53
2 5, 7, 15, 17, 19, 20, 22, 27, 33, 34, 38, 47, 48, 50, 53
3 3, 4, 9, 12, 13, 15, 17, 19, 22, 29, 31, 32, 33, 34, 36, 38, 46, 47, 53
Table A.11: Extracted features obtained for each subset, by performing SFS with the third set of featureselection criteria, with 5-fold cross-validation.
Subset Features
Sensitivity1 5, 9, 15, 19, 20, 28, 33, 42, 44, 47, 48, 49, 50
2 3, 9, 12, 13, 14, 17, 19, 20, 27, 33, 35, 37, 47, 49, 50
3 3, 9, 12, 17, 24, 27, 33, 38, 43, 48, 49, 50
Specificity1 3, 8, 9, 11, 13, 15, 18, 21, 22, 25, 26, 29, 30, 33, 47, 49
2 3, 8, 9, 12, 13, 17, 18, 19, 22, 28, 33, 34, 37, 39, 47, 49
3 1, 3, 8, 12, 13, 14, 15, 18, 22, 26, 29, 30, 33, 47, 49
A5
Table A.12: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the first set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.76 ± 0.02 0.76 ± 0.03 0.76 ± 0.02
Sensitivity subset 1 Accuracy 0.69 ± 0.03 0.70 ± 0.03 0.68 ± 0.03
and specificity subset 1 Sensitivity 0.68 ± 0.05 0.70 ± 0.04 0.67 ± 0.05
Specificity 0.70 ± 0.05 0.68 ± 0.05 0.72 ± 0.05
AUC 0.77 ± 0.02 0.76 ± 0.01 0.77 ± 0.01
Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.70 ± 0.02 0.69 ± 0.02
and specificity subset 2 Sensitivity 0.69 ± 0.03 0.70 ± 0.03 0.68 ± 0.03
Specificity 0.70 ± 0.03 0.69 ± 0.03 0.71 ± 0.03
AUC 0.77 ± 0.02 0.77 ± 0.01 0.77 ± 0.01
Sensitivity subset 1 Accuracy 0.70 ± 0.02 0.71 ± 0.02 0.68 ± 0.02
and specificity subset 3 Sensitivity 0.69 ± 0.03 0.71 ± 0.03 0.67 ± 0.03
Specificity 0.71 ± 0.03 0.69 ± 0.03 0.73 ± 0.03
AUC 0.77 ± 0.02 0.76 ± 0.02 0.77 ± 0.02
Sensitivity subset 2 Accuracy 0.69 ± 0.02 0.71 ± 0.02 0.68 ± 0.02
and specificity subset 1 Sensitivity 0.69 ± 0.04 0.72 ± 0.03 0.66 ± 0.04
Specificity 0.70 ± 0.04 0.67 ± 0.04 0.73 ± 0.04
AUC 0.77 ± 0.02 0.76 ± 0.02 0.77 ± 0.02
Sensitivity subset 2 Accuracy 0.70 ± 0.03 0.71 ± 0.03 0.68 ± 0.03
and specificity subset 2 Sensitivity 0.70 ± 0.05 0.72 ± 0.04 0.67 ± 0.05
Specificity 0.69 ± 0.04 0.65 ± 0.04 0.70 ± 0.04
AUC 0.76 ± 0.02 0.76 ± 0.02 0.77 ± 0.02
Sensitivity subset 2 Accuracy 0.70 ± 0.03 0.71 ± 0.03 0.68 ± 0.03
and specificity subset 3 Sensitivity 0.69 ± 0.04 0.72 ± 0.04 0.66 ± 0.05
Specificity 0.70 ± 0.03 0.67 ± 0.04 0.73 ± 0.03
AUC 0.77 ± 0.02 0.76 ± 0.02 0.77 ± 0.02
Sensitivity subset 3 Accuracy 0.69 ± 0.03 0.71 ± 0.03 0.68 ± 0.03
and specificity subset 1 Sensitivity 0.69 ± 0.05 0.72 ± 0.04 0.66 ± 0.05
Specificity 0.70 ± 0.03 0.67 ± 0.03 0.73 ± 0.03
AUC 0.77 ± 0.03 0.76 ± 0.03 0.77 ± 0.03
Sensitivity subset 3 Accuracy 0.69 ± 0.03 0.70 ± 0.03 0.68 ± 0.03
and specificity subset 2 Sensitivity 0.69 ± 0.04 0.71 ± 0.04 0.66 ± 0.04
Specificity 0.70 ± 0.03 0.67 ± 0.03 0.70 ± 0.03
AUC 0.77 ± 0.01 0.76 ± 0.01 0.77 ± 0.01
Sensitivity subset 3 Accuracy 0.70 ± 0.01 0.71 ± 0.01 0.68 ± 0.01
and specificity subset 3 Sensitivity 0.70 ± 0.02 0.72 ± 0.02 0.67 ± 0.02
Specificity 0.70 ± 0.03 0.68 ± 0.03 0.73 ± 0.03
A6
Table A.13: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the second set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.76 ± 0.01 0.75 ± 0.02 0.76 ± 0.01
Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.70 ± 0.02 0.67 ± 0.02
and specificity subset 1 Sensitivity 0.69 ± 0.03 0.71 ± 0.03 0.65 ± 0.03
Specificity 0.69 ± 0.03 0.67 ± 0.03 0.72 ± 0.03
AUC 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02
Sensitivity subset 1 Accuracy 0.68 ± 0.02 0.69 ± 0.02 0.67 ± 0.03
and specificity subset 2 Sensitivity 0.68 ± 0.04 0.70 ± 0.04 0.64 ± 0.04
Specificity 0.70 ± 0.03 0.67 ± 0.03 0.73 ± 0.03
AUC 0.76 ± 0.01 0.76 ± 0.01 0.76 ± 0.01
Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.71 ± 0.02 0.67 ± 0.02
and specificity subset 3 Sensitivity 0.68 ± 0.03 0.71 ± 0.03 0.65 ± 0.04
Specificity 0.71 ± 0.03 0.68 ± 0.03 0.73 ± 0.03
AUC 0.74 ± 0.02 0.74 ± 0.02 0.75 ± 0.02
Sensitivity subset 2 Accuracy 0.68 ± 0.03 0.70 ± 0.03 0.67 ± 0.03
and specificity subset 1 Sensitivity 0.68 ± 0.04 0.71 ± 0.04 0.64 ± 0.05
Specificity 0.68 ± 0.04 0.64 ± 0.04 0.72 ± 0.03
AUC 0.74 ± 0.02 0.74 ± 0.02 0.74 ± 0.02
Sensitivity subset 2 Accuracy 0.68 ± 0.03 0.69 ± 0.03 0.66 ± 0.03
and specificity subset 2 Sensitivity 0.67 ± 0.05 0.71 ± 0.05 0.63 ± 0.05
Specificity 0.69 ± 0.04 0.65 ± 0.04 0.72 ± 0.03
AUC 0.75 ± 0.02 0.75 ± 0.02 0.76 ± 0.02
Sensitivity subset 2 Accuracy 0.69 ± 0.02 0.70 ± 0.02 0.67 ± 0.03
and specificity subset 3 Sensitivity 0.69 ± 0.04 0.72 ± 0.03 0.64 ± 0.04
Specificity 0.69 ± 0.04 0.65 ± 0.05 0.74 ± 0.04
AUC 0.75 ± 0.02 0.75 ± 0.02 0.76 ± 0.02
Sensitivity subset 3 Accuracy 0.69 ± 0.03 0.70 ± 0.03 0.67 ± 0.03
and specificity subset 1 Sensitivity 0.68 ± 0.03 0.72 ± 0.03 0.64 ± 0.04
Specificity 0.69 ± 0.02 0.65 ± 0.03 0.73 ± 0.03
AUC 0.75 ± 0.02 0.74 ± 0.02 0.75 ± 0.02
Sensitivity subset 3 Accuracy 0.69 ± 0.03 0.70 ± 0.03 0.67 ± 0.03
and specificity subset 2 Sensitivity 0.69 ± 0.05 0.72 ± 0.04 0.65 ± 0.05
Specificity 0.69 ± 0.04 0.65 ± 0.04 0.73 ± 0.04
AUC 0.76 ± 0.02 0.76 ± 0.02 0.76 ± 0.02
Sensitivity subset 3 Accuracy 0.69 ± 0.02 0.71 ± 0.02 0.68 ± 0.03
and specificity subset 3 Sensitivity 0.69 ± 0.03 0.72 ± 0.03 0.65 ± 0.04
Specificity 0.70 ± 0.03 0.67 ± 0.03 0.73 ± 0.03
A7
Table A.14: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the third set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.73 ± 0.02 0.73 ± 0.02 0.74 ± 0.02
Sensitivity subset 1 Accuracy 0.68 ± 0.03 0.69 ± 0.02 0.66 ± 0.03
and specificity subset 1 Sensitivity 0.68 ± 0.04 0.71 ± 0.04 0.63 ± 0.05
Specificity 0.68 ± 0.04 0.64 ± 0.04 0.73 ± 0.04
AUC 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02
Sensitivity subset 1 Accuracy 0.69 ± 0.03 0.71 ± 0.02 0.67 ± 0.03
and specificity subset 2 Sensitivity 0.69 ± 0.03 0.72 ± 0.03 0.64 ± 0.04
Specificity 0.70 ± 0.03 0.66 ± 0.03 0.74 ± 0.03
AUC 0.73 ± 0.02 0.73 ± 0.02 0.74 ± 0.02
Sensitivity subset 1 Accuracy 0.67 ± 0.03 0.69 ± 0.03 0.65 ± 0.03
and specificity subset 3 Sensitivity 0.67 ± 0.05 0.71 ± 0.04 0.62 ± 0.05
Specificity 0.68 ± 0.04 0.64 ± 0.04 0.73 ± 0.04
AUC 0.74 ± 0.02 0.73 ± 0.02 0.74 ± 0.02
Sensitivity subset 2 Accuracy 0.68 ± 0.02 0.70 ± 0.02 0.66 ± 0.02
and specificity subset 1 Sensitivity 0.68 ± 0.04 0.71 ± 0.03 0.64 ± 0.04
Specificity 0.68 ± 0.03 0.64 ± 0.04 0.73 ± 0.03
AUC 0.74 ± 0.02 0.74 ± 0.02 0.74 ± 0.02
Sensitivity subset 2 Accuracy 0.69 ± 0.02 0.70 ± 0.03 0.67 ± 0.02
and specificity subset 2 Sensitivity 0.68 ± 0.04 0.71 ± 0.04 0.65 ± 0.03
Specificity 0.70 ± 0.03 0.66 ± 0.03 0.73 ± 0.03
AUC 0.73 ± 0.02 0.73 ± 0.02 0.74 ± 0.02
Sensitivity subset 2 Accuracy 0.68 ± 0.03 0.69 ± 0.03 0.66 ± 0.04
and specificity subset 3 Sensitivity 0.67 ± 0.05 0.71 ± 0.05 0.63 ± 0.06
Specificity 0.68 ± 0.04 0.64 ± 0.03 0.73 ± 0.04
AUC 0.75 ± 0.02 0.74 ± 0.02 0.75 ± 0.02
Sensitivity subset 3 Accuracy 0.69 ± 0.03 0.70 ± 0.02 0.66 ± 0.03
and specificity subset 1 Sensitivity 0.68 ± 0.04 0.73 ± 0.03 0.63 ± 0.04
Specificity 0.69 ± 0.04 0.64 ± 0.04 0.74 ± 0.04
AUC 0.75 ± 0.02 0.75 ± 0.02 0.75 ± 0.02
Sensitivity subset 3 Accuracy 0.69 ± 0.03 0.71 ± 0.02 0.67 ± 0.02
and specificity subset 2 Sensitivity 0.68 ± 0.03 0.72 ± 0.03 0.64 ± 0.04
Specificity 0.70 ± 0.03 0.66 ± 0.03 0.74 ± 0.04
AUC 0.74 ± 0.02 0.74 ± 0.02 0.75 ± 0.02
Sensitivity subset 3 Accuracy 0.68 ± 0.03 0.70 ± 0.03 0.66 ± 0.03
and specificity subset 3 Sensitivity 0.68 ± 0.05 0.72 ± 0.04 0.63 ± 0.05
Specificity 0.68 ± 0.05 0.64 ± 0.05 0.73 ± 0.05
A8
Appendix B
Discharge Day Evaluation
In this section are presented the extended results obtained for the different approaches implemented,
using only the discharge day samples in the test set. In Section B.1 are presented the results obtained
for the single models with prime features, in Section B.2 the results obtained for the single models with
extracted features and in Section B.3 the results obtained for the multimodels with extracted features.
B.1 Single Model with Prime Features
Table B.1: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with extracted features, varying the number of clusters between 2
and 10 and the degree of fuzziness between 1.1 and 1.5.
Degree of Fuzziness
1.1 1.2 1.3 1.4 1.5
2 0.63 ± 0.05 0.64 ± 0.06 0.64 ± 0.06 0.63 ± 0.07 0.62 ± 0.05
3 0.59 ± 0.07 0.62 ± 0.08 0.64 ± 0.07 0.66 ± 0.06 0.64 ± 0.06
4 0.61 ± 0.07 0.61 ± 0.08 0.65 ± 0.06 0.64 ± 0.07 0.65 ± 0.06
Number 5 0.63 ± 0.06 0.59 ± 0.08 0.59 ± 0.09 0.63 ± 0.05 0.64 ± 0.06
of 6 0.64 ± 0.08 0.61 ± 0.08 0.63 ± 0.07 0.62 ± 0.07 0.63 ± 0.06
Culsters 7 0.62 ± 0.07 0.60 ± 0.07 0.65 ± 0.07 0.60 ± 0.09 0.63 ± 0.06
8 0.63 ± 0.06 0.64 ± 0.08 0.61 ± 0.07 0.63 ± 0.06 0.62 ± 0.06
9 0.61 ± 0.08 0.62 ± 0.08 0.63 ± 0.08 0.61 ± 0.07 0.58 ± 0.06
10 0.61 ± 0.06 0.65 ± 0.06 0.61 ± 0.05 0.58 ± 0.08 0.61 ± 0.05
B1
Table B.2: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with prime features, varying the number of clusters between 2 and 10
and the degree of fuzziness between 1.6 and 2.
Degree of Fuzziness
1.6 1.7 1.8 1.9 2
2 0.61 ± 0.06 0.61 ± 0.06 0.61 ± 0.05 0.62 ± 0.04 0.62 ± 0.04
3 0.63 ± 0.05 0.62 ± 0.05 0.61 ± 0.04 0.60 ± 0.04 0.61 ± 0.05
4 0.63 ± 0.06 0.62 ± 0.04 0.62 ± 0.06 0.63 ± 0.06 0.62 ± 0.06
Number 5 0.63 ± 0.06 0.63 ± 0.06 0.63 ± 0.06 0.63 ± 0.07 0.63 ± 0.06
of 6 0.64 ± 0.06 0.63 ± 0.07 0.63 ± 0.07 0.62 ± 0.06 0.63 ± 0.06
Clusters 7 0.63 ± 0.06 0.63 ± 0.07 0.63 ± 0.07 0.64 ± 0.07 0.64 ± 0.07
8 0.64 ± 0.07 0.63 ± 0.07 0.63 ± 0.06 0.64 ± 0.07 0.64 ± 0.06
9 0.63 ± 0.07 0.63 ± 0.07 0.63 ± 0.06 0.63 ± 0.07 0.65 ± 0.07
10 0.63 ± 0.06 0.63 ± 0.07 0.63 ± 0.08 0.64 ± 0.06 0.63 ± 0.06
Table B.3: Prime features obtained for each subset, after SFS with 5-fold cross-validation.
Subset Features
1 3, 4, 5, 7, 8
2 3, 4, 5
3 3, 4, 5, 8
B.2 Single Model with Extracted Features
Table B.4: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with extracted features, varying the number of clusters between 2
and 10 and the degree of fuzziness between 1.1 and 1.5.
Degree of Fuzziness
1.1 1.2 1.3 1.4 1.5
2 0.61 ± 0.07 0.62 ± 0.08 0.61 ± 0.05 0.63 ± 0.07 0.62 ± 0.08
3 0.61 ± 0.10 0.64 ± 0.07 0.62 ± 0.08 0.64 ± 0.07 0.64 ± 0.06
4 0.66 ± 0.06 0.62 ± 0.07 0.64 ± 0.06 0.62 ± 0.06 0.61 ± 0.06
Number 5 0.64 ± 0.08 0.65 ± 0.04 0.63 ± 0.09 0.64 ± 0.04 0.61 ± 0.04
of 6 0.61 ± 0.11 0.58 ± 0.06 0.61 ± 0.10 0.59 ± 0.08 0.58 ± 0.07
Culsters 7 0.60 ± 0.09 0.63 ± 0.07 0.61 ± 0.07 0.59 ± 0.04 0.55 ± 0.05
8 0.58 ± 0.09 0.61 ± 0.05 0.62 ± 0.06 0.59 ± 0.06 0.57 ± 0.03
9 0.61 ± 0.10 0.60 ± 0.09 0.59 ± 0.05 0.56 ± 0.05 0.58 ± 0.08
10 0.60 ± 0.09 0.58 ± 0.08 0.59 ± 0.04 0.55 ± 0.03 0.57 ± 0.08
B2
Table B.5: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the FS dataset with extracted features, varying the number of clusters between 2
and 10 and the degree of fuzziness between 1.6 and 2.
Degree of Fuzziness
1.6 1.7 1.8 1.9 2
2 0.63 ± 0.06 0.62 ± 0.07 0.63 ± 0.07 0.63 ± 0.07 0.63 ± 0.07
3 0.63 ± 0.06 0.62 ± 0.05 0.62 ± 0.06 0.61 ± 0.06 0.63 ± 0.06
4 0.61 ± 0.06 0.61 ± 0.06 0.61 ± 0.07 0.62 ± 0.05 0.62 ± 0.07
Number 5 0.60 ± 0.07 0.60 ± 0.08 0.61 ± 0.05 0.61 ± 0.06 0.61 ± 0.06
of 6 0.59 ± 0.06 0.61 ± 0.06 0.62 ± 0.06 0.60 ± 0.05 0.61 ± 0.07
Clusters 7 0.58 ± 0.06 0.59 ± 0.05 0.59 ± 0.05 0.61 ± 0.06 0.61 ± 0.08
8 0.58 ± 0.07 0.59 ± 0.05 0.58 ± 0.07 0.59 ± 0.06 0.57 ± 0.06
9 0.58 ± 0.04 0.59 ± 0.06 0.59 ± 0.04 0.60 ± 0.03 0.58 ± 0.03
10 0.58 ± 0.05 0.56 ± 0.04 0.58 ± 0.04 0.58 ± 0.05 0.57 ± 0.04
Table B.6: Extracted features obtained for each subset, after SFS with 5-fold cross-validation.
Subset Features
1 4, 6, 7, 9, 10, 12, 13, 14, 15, 18, 22, 23, 24, 26, 31, 36, 41, 45, 46, 51
2 3, 12, 22, 26, 51
3 3, 8, 23, 24, 49, 50
B.3 Multimodel with Extracted Features
Table B.7: Extracted features obtained for each subset, by performing SFS with the first set of featureselection criteria, with 5-fold cross-validation.
Subset Features
Sensitivity1 9, 11, 18, 21, 22, 23, 24, 28, 30, 31, 41, 45, 46, 48, 49, 50, 51, 52
2 14, 19, 25, 30, 31, 36, 40, 42, 43, 44, 50, 51
3 5, 10, 13, 15, 18, 21, 23, 28, 31, 41, 48, 51, 52
Specificity1 1, 8, 10, 15, 16, 17, 19, 22, 24, 25, 31, 33, 35, 36, 38, 42, 45, 46, 47, 49, 51
2 3, 7, 14, 16, 17, 19, 21, 22, 24, 32, 33, 36, 46, 47, 51, 52
3 1, 8, 15, 16, 17, 20, 21, 22, 29, 33, 35, 41, 42, 47, 48, 49, 51, 52
B3
Table B.8: Extracted features obtained for each subset, by performing SFS with the second set offeature selection criteria, with 5-fold cross-validation.
Subset Features
Sensitivity1 5, 8, 11, 12, 14, 20, 24, 30, 31, 40, 46, 48, 49, 51, 52
2 3, 5, 6, 7, 10, 12, 15, 16, 19, 23, 26, 31, 32, 36, 38, 43, 44, 45, 48, 49
3 3, 5, 6, 10, 18, 22, 23, 24, 25, 27, 30, 31, 41, 44, 49, 51
Specificity1 7, 10, 15, 20, 23, 24, 29, 31, 41, 44, 46, 49, 51, 53
2 10, 13, 18, 20, 22, 24, 26, 36, 46, 51
3 5, 6, 7, 10, 18, 26, 30, 31, 43, 49, 52
Table B.9: Extracted features obtained for each subset, by performing SFS with the third set of featureselection criteria, with 5-fold cross-validation.
Subset Features
Sensitivity1 3, 6, 9, 11, 13, 24, 31, 32, 38, 48, 49, 51, 52
2 5, 6, 9, 18, 20, 38, 49, 50
3 9, 10, 12, 13, 18, 26, 30, 37, 49, 50
Specificity1 3, 6, 7, 8, 9, 12, 13, 14, 15, 18, 21, 22, 23, 24, 25, 26, 31, 32, 38, 41, 45, 49, 50, 52
2 3, 10, 13, 14, 15, 18, 20, 22, 23, 24, 26, 27, 30, 31, 34, 36, 38, 41, 44, 49, 50, 52
3 5, 8, 11, 12, 13, 15, 24, 25, 26, 27, 31, 37, 41, 48, 49, 50
B4
Table B.10: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the first set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.67 ± 0.09 0.66 ± 0.09 0.67 ± 0.09
Sensitivity subset 1 Accuracy 0.69 ± 0.02 0.63 ± 0.02 0.73 ± 0.02
and specificity subset 1 Sensitivity 0.61 ± 0.12 0.64 ± 0.12 0.55 ± 0.10
Specificity 0.69 ± 0.03 0.63 ± 0.03 0.75 ± 0.03
AUC 0.67 ± 0.09 0.67 ± 0.09 0.67 ± 0.09
Sensitivity subset 1 Accuracy 0.68 ± 0.03 0.64 ± 0.04 0.73 ± 0.03
and specificity subset 2 Sensitivity 0.59 ± 0.16 0.64 ± 0.15 0.49 ± 0.19
Specificity 0.69 ± 0.04 0.64 ± 0.04 0.74 ± 0.03
AUC 0.65 ± 0.11 0.65 ± 0.10 0.65 ± 0.11
Sensitivity subset 1 Accuracy 0.68 ± 0.03 0.63 ± 0.03 0.73 ± 0.04
and specificity subset 3 Sensitivity 0.58 ± 0.16 0.62 ± 0.18 0.49 ± 0.13
Specificity 0.69 ± 0.03 0.63 ± 0.04 0.75 ± 0.04
AUC 0.65 ± 0.09 0.64 ± 0.09 0.65 ± 0.08
Sensitivity subset 2 Accuracy 0.67 ± 0.04 0.61 ± 0.04 0.72 ± 0.05
and specificity subset 1 Sensitivity 0.55 ± 0.14 0.59 ± 0.13 0.52 ± 0.13
Specificity 0.67 ± 0.04 0.61 ± 0.04 0.74 ± 0.05
AUC 0.66 ± 0.11 0.66 ± 0.12 0.67 ± 0.11
Sensitivity subset 2 Accuracy 0.67 ± 0.04 0.63 ± 0.04 0.72 ± 0.04
and specificity subset 2 Sensitivity 0.60 ± 0.18 0.63 ± 0.19 0.49 ± 0.17
Specificity 0.68 ± 0.04 0.62 ± 0.04 0.74 ± 0.04
AUC 0.66 ± 0.08 0.65 ± 0.08 0.66 ± 0.08
Sensitivity subset 2 Accuracy 0.67 ± 0.03 0.62 ± 0.03 0.73 ± 0.03
and specificity subset 3 Sensitivity 0.60 ± 0.12 0.63 ± 0.12 0.55 ± 0.12
Specificity 0.68 ± 0.03 0.62 ± 0.03 0.74 ± 0.03
AUC 0.67 ± 0.07 0.65 ± 0.08 0.68 ± 0.07
Sensitivity subset 3 Accuracy 0.67 ± 0.03 0.62 ± 0.03 0.73 ± 0.04
and specificity subset 1 Sensitivity 0.54 ± 0.12 0.58 ± 0.14 0.52 ± 0.13
Specificity 0.68 ± 0.04 0.62 ± 0.04 0.75 ± 0.04
AUC 0.69 ± 0.08 0.69 ± 0.08 0.70 ± 0.08
Sensitivity subset 3 Accuracy 0.68 ± 0.04 0.63 ± 0.04 0.73 ± 0.03
and specificity subset 2 Sensitivity 0.61 ± 0.12 0.65 ± 0.12 0.54 ± 0.14
Specificity 0.68 ± 0.04 0.63 ± 0.04 0.75 ± 0.04
AUC 0.67 ± 0.10 0.67 ± 0.11 0.68 ± 0.10
Sensitivity subset 3 Accuracy 0.67 ± 0.04 0.63 ± 0.04 0.72 ± 0.04
and specificity subset 3 Sensitivity 0.57 ± 0.14 0.62 ± 0.14 0.54 ± 0.15
Specificity 0.68 ± 0.04 0.63 ± 0.04 0.73 ± 0.04
B5
Table B.11: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the second set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.67 ± 0.10 0.66 ± 0.04 0.66 ± 0.09
Sensitivity subset 1 Accuracy 0.64 ± 0.03 0.61 ± 0.04 0.69 ± 0.04
and specificity subset 1 Sensitivity 0.61 ± 0.18 0.67 ± 0.15 0.53 ± 0.16
Specificity 0.65 ± 0.04 0.61 ± 0.04 0.71 ± 0.04
AUC 0.67 ± 0.09 0.67 ± 0.09 0.67 ± 0.08
Sensitivity subset 1 Accuracy 0.66 ± 0.04 0.62 ± 0.04 0.69 ± 0.04
and specificity subset 2 Sensitivity 0.60 ± 0.17 0.66 ± 0.16 0.51 ± 0.17
Specificity 0.66 ± 0.04 0.61 ± 0.04 0.71 ± 0.05
AUC 0.68 ± 0.08 0.67 ± 0.08 0.68 ± 0.08
Sensitivity subset 1 Accuracy 0.64 ± 0.05 0.61 ± 0.05 0.69 ± 0.05
and specificity subset 3 Sensitivity 0.62 ± 0.14 0.64 ± 0.15 0.57 ± 0.13
Specificity 0.63 ± 0.05 0.61 ± 0.05 0.70 ± 0.05
AUC 0.67 ± 0.08 0.67 ± 0.08 0.68 ± 0.08
Sensitivity subset 2 Accuracy 0.65 ± 0.05 0.61 ± 0.05 0.70 ± 0.05
and specificity subset 1 Sensitivity 0.59 ± 0.16 0.63 ± 0.16 0.51 ± 0.17
Specificity 0.66 ± 0.05 0.61 ± 0.05 0.72 ± 0.06
AUC 0.69 ± 0.05 0.69 ± 0.05 0.68 ± 0.05
Sensitivity subset 2 Accuracy 0.66 ± 0.03 0.61 ± 0.03 0.72 ± 0.03
and specificity subset 2 Sensitivity 0.62 ± 0.10 0.67 ± 0.08 0.50 ± 0.11
Specificity 0.66 ± 0.03 0.61 ± 0.03 0.73 ± 0.03
AUC 0.68 ± 0.08 0.67 ± 0.08 0.69 ± 0.08
Sensitivity subset 2 Accuracy 0.65 ± 0.04 0.62 ± 0.04 0.69 ± 0.04
and specificity subset 3 Sensitivity 0.61 ± 0.15 0.64 ± 0.15 0.52 ± 0.14
Specificity 0.65 ± 0.04 0.62 ± 0.04 0.70 ± 0.04
AUC 0.68 ± 0.06 0.67 ± 0.06 0.68 ± 0.06
Sensitivity subset 3 Accuracy 0.66 ± 0.02 0.64 ± 0.04 0.70 ± 0.03
and specificity subset 1 Sensitivity 0.62 ± 0.12 0.65 ± 0.12 0.57 ± 0.14
Specificity 0.66 ± 0.03 0.63 ± 0.03 0.70 ± 0.04
AUC 0.69 ± 0.06 0.68 ± 0.06 0.69 ± 0.06
Sensitivity subset 3 Accuracy 0.65 ± 0.05 0.62 ± 0.04 0.68 ± 0.05
and specificity subset 2 Sensitivity 0.59 ± 0.11 0.62 ± 0.10 0.55 ± 0.11
Specificity 0.65 ± 0.05 0.62 ± 0.05 0.69 ± 0.05
AUC 0.70 ± 0.10 0.69 ± 0.10 0.71 ± 0.09
Sensitivity subset 3 Accuracy 0.65 ± 0.04 0.62 ± 0.04 0.69 ± 0.04
and specificity subset 3 Sensitivity 0.67 ± 0.12 0.71 ± 0.12 0.61 ± 0.14
Specificity 0.65 ± 0.04 0.61 ± 0.04 0.70 ± 0.04
B6
Table B.12: Mean and standard deviation of the results obtained after performing 5 × 5-foldcross-validation in the MA dataset for the third set of feature selection criteria.
Weighted distance Maximum distance Average
AUC 0.67 ± 0.07 0.66 ± 0.04 0.68 ± 0.07
Sensitivity subset 1 Accuracy 0.67 ± 0.04 0.63 ± 0.04 0.71 ± 0.04
and specificity subset 1 Sensitivity 0.56 ± 0.14 0.59 ± 0.15 0.52 ± 0.14
Specificity 0.67 ± 0.04 0.63 ± 0.04 0.72 ± 0.04
AUC 0.64 ± 0.12 0.64 ± 0.12 0.65 ± 0.12
Sensitivity subset 1 Accuracy 0.66 ± 0.04 0.62 ± 0.05 0.72 ± 0.04
and specificity subset 2 Sensitivity 0.55 ± 0.15 0.60 ± 0.14 0.48 ± 0.16
Specificity 0.67 ± 0.04 0.62 ± 0.05 0.73 ± 0.04
AUC 0.68 ± 0.08 0.68 ± 0.06 0.68 ± 0.07
Sensitivity subset 1 Accuracy 0.66 ± 0.04 0.62 ± 0.03 0.71 ± 0.03
and specificity subset 3 Sensitivity 0.59 ± 0.14 0.65 ± 0.12 0.52 ± 0.12
Specificity 0.66 ± 0.04 0.62 ± 0.03 0.72 ± 0.04
AUC 0.67 ± 0.08 0.66 ± 0.08 0.67 ± 0.07
Sensitivity subset 2 Accuracy 0.66 ± 0.03 0.62 ± 0.03 0.71 ± 0.02
and specificity subset 1 Sensitivity 0.60 ± 0.16 0.66 ± 0.14 0.53 ± 0.15
Specificity 0.67 ± 0.03 0.62 ± 0.04 0.72 ± 0.03
AUC 0.65 ± 0.10 0.64 ± 0.09 0.65 ± 0.09
Sensitivity subset 2 Accuracy 0.65 ± 0.03 0.61 ± 0.03 0.70 ± 0.04
and specificity subset 2 Sensitivity 0.61 ± 0.16 0.66 ± 0.15 0.50 ± 0.19
Specificity 0.66 ± 0.04 0.61 ± 0.04 0.71 ± 0.04
AUC 0.63 ± 0.09 0.63 ± 0.09 0.62 ± 0.09
Sensitivity subset 2 Accuracy 0.63 ± 0.04 0.59 ± 0.04 0.67 ± 0.04
and specificity subset 3 Sensitivity 0.56 ± 0.18 0.62 ± 0.19 0.45 ± 0.18
Specificity 0.64 ± 0.04 0.59 ± 0.04 0.69 ± 0.05
AUC 0.66 ± 0.09 0.65 ± 0.09 0.67 ± 0.09
Sensitivity subset 3 Accuracy 0.66 ± 0.04 0.64 ± 0.04 0.70 ± 0.04
and specificity subset 1 Sensitivity 0.56 ± 0.17 0.63 ± 0.16 0.51 ± 0.14
Specificity 0.67 ± 0.04 0.64 ± 0.04 0.71 ± 0.04
AUC 0.67 ± 0.06 0.67 ± 0.06 0.68 ± 0.06
Sensitivity subset 3 Accuracy 0.67 ± 0.03 0.64 ± 0.03 0.71 ± 0.03
and specificity subset 2 Sensitivity 0.59 ± 0.10 0.66 ± 0.11 0.53 ± 0.11
Specificity 0.67 ± 0.03 0.64 ± 0.03 0.72 ± 0.04
AUC 0.67 ± 0.07 0.67 ± 0.07 0.67 ± 0.07
Sensitivity subset 3 Accuracy 0.64 ± 0.04 0.61 ± 0.04 0.68 ± 0.04
and specificity subset 3 Sensitivity 0.58 ± 0.15 0.65 ± 0.17 0.50 ± 0.17
Specificity 0.64 ± 0.04 0.61 ± 0.04 0.70 ± 0.04
7