Securing Critical Infrastructures at
software and interdependency levels
George Stergiopoulos
November 2015
ii
Securing Critical Infrastructures
at software and interdependency levels
George Stergiopoulos
A dissertation submitted for the partial fulfillment
of a Ph.D. degree
November 2015
Department of Informatics
Athens University of Economics & Business
Athens, Greece
iii
Supervising Committee:
1. Dimitris Gritzalis, Professor, Athens University of Economics & Business (Chair)
2. Ioannis Mavridis, Associate Professor, University of Macedonia
3. Panagiotis Katsaros, Assistant Professor, Aristotle University of Thessaloniki
Examination Committee:
1. Dimitris Gritzalis, Professor, Athens University of Economics & Business
2. Theodoros Apostolopoulos, Professor, Athens University of Economics & Business
3. Ioannis Mavridis, Associate Professor, University of Macedonia
4. Konstantinos Lambrinoudakis, Associate Professor, University of Piraeus
5. Panagiotis Katsaros, Assistant Professor, Aristotle University of Thessaloniki
6. Ioannis Marias, Assistant Professor, Athens University of Economics & Business
7. Panos Kotzanikolaou, Assistant Professor, University of Piraeus
iv
Securing Critical Infrastructures
at software and interdependency levels
Copyright © 2015
by
George Stergiopoulos
Department of Informatics
Athens University of Economics and Business
76 Patission Ave., Athens GR-10434, Greece
v
All rights reserved. No part of this manuscript may be reproduced or transmitted
in any form or by any means, electronic, mechanical, photocopying, recording, or
otherwise, without the prior written permission of the author.
"Η έγκριση διδακτορικής διατριβής υπό του Τμήματος Πληροφορικής του Οικονομικού
Πανεπιστημίου Αθηνών δεν υποδηλοί αποδοχή των γνωμών του συγγραφέως.”
(Ν. 5343/ 1932, άρθρο. 202)
vi
Acknowledgements
It is common that doctoral students overlook the acknowledgements section. Instead, they
focus their writing on the main content while trying to produce a robust and all around sound
text that will adequately depict their research efforts. Still, most fail to recognize that, without
the significant help of large group of people, these research efforts would have borne little to
no fruit. This is why, I will dedicate the following paragraphs to all those that helped me,
directly or indirectly, achieve my goal during my doctoral studies.
First and foremost, I would like to thank my Ph.D. supervisor Prof. Dimitris Gritzalis. His
role was twofold, that of an academic teacher and a personal research and business advisor.
Thus, he was influential to my research. Prof. Gritzalis guided me through my doctoral studies
and helped me mature both as a researcher and as a professional. I am grateful to be a part of
Prof. Dimitris Gritzalis’s Critical Infrastructure Protection group; a group, which, under Prof.
Gritzalis’s constant care and supervision, infused me with a wide variety of knowledge,
experience and understanding of both the academic, and the business sector. He is the person
who motivated my decision to focus on critical infrastructure protection and information
systems analysis for the past 5 years.
I also would like to thank Ass. Prof. Panagiotis Kotzanikolaou and Ass. Prof. Panagiotis
Katsaros for their excellent contribution and help during my doctoral studies. Their experience,
patience and guidance was of utmost importance and acted as a steady light that helped me
overcome multiple obstacles during my research efforts; both technical and theoretical. I would
also like to thank Bill Tsoumas who believed in me from the very start and altruistically helped
during my preliminary steps in research.
During my doctoral studies, I was blessed to have colleagues that supported me in many
ways. Thus, I would like to thank Dr. Marianthi Theoharidou and Dr. Bill Tsoumas for teaching
me how to write a paper. I was lucky enough to have Marianthi as my partner in some of my
research efforts. I am also grateful to Nikos Virvilis and Miltiadis Kandias for their friendship,
cooperation and support. Their help and guidance bore fruits and aided me to learn how to
manage my tasks and complete them flawlessly and on time. Last but not least, I would like to
thank certain people who I met during my doctoral studies and influenced my course of action.
I sincerely thank Alexios Mylonas, Nikos Tsalis, Vasilis Stavrou and Nick Bozovits for their
help, cooperation and friendship during all those years in the university. I would also be remiss
not to thank Vasilis Spyropoulos, Vasilis Zafeiris, Nantia Makrinioti and Tassos Venetis, for
their friendship and for the challenges that we faced together while working in the Information
Systems and Databases lab (ISLAB).
Partners and academic staff aside, I feel obliged to express my gratitude to other people that
influenced my research journey, supported and aided me when times were harsh and I felt that
I would never make it. People that are a part of who I am now, and, thus owe them a lot.
vii
Amongst many, I am deeply thankful to Violetta Sotiropoulou, Mihalis Maniatakos and George
Kourepis. Their friendship and support went the extra mile in mitigating anxiety, difficulties
and ill thoughts that occurred along the way. I am very grateful to have them by my side and
would like to thank them for helping me mature as a personality.
Finally, I feel obliged to express my deepest gratitude and love to my parents for their
unconditional, constant love and support; both psychological and economical. I thank them for
infusing me the ethics, qualities and understanding needed to succeed in life. For this reason,
this dissertation is dedicated to them.
Athens, 5th September 2015
viii
Dedication
To my parents:
“When I was a boy, my parents were so ignorant I could hardly stand to
have them around. But when I got to be thirty, I was astonished at how
much they have learned in just a few years”
ix
Abstract
A Critical infrastructure is the backbone of a nation's economy, security and health. It is
those infrastructures that provide power and water to homes, support the transportation and
communication systems people rely on. The criteria for determining what might be a critical
infrastructure, and which infrastructures thus qualify, have expanded over time (DHS, 2013).
A Critical infrastructure is defined as those assets, systems and networks, whether physical or
virtual, so vital to a country that their incapacitation or destruction would have a debilitating
effect on security, national economic security, national public health or safety, or any
combination thereof (DHS, 2013).
At the very least, a growing list of infrastructures in need of protection will require the
federal government to prioritize its efforts and try to minimize the impact on the nation’s critical
infrastructures of any future failure of any kind (e.g. terrorist attack or systems failure) while
taking into account what those impacts might be and the likelihood of their occurring (Motef
et al., 2003).
Considering all the above, it is made obvious that critical infrastructures, along with their
services and systems must be protected against all types of failures; both human-made and
natural phenomena. Critical infrastructures provide services needed for a nation to function
properly and support its citizens, such as the health care system, transportations,
communications etc. Even more, failures in these infrastructures can be triggered by attackers
in order to maim a nation and/or to increase revenues (e.g. theft, information leakage etc).
Up until now, research has focused in securing critical infrastructures by utilizing Risk
Assessment methodologies based on ISOs (like the ISO 27001), security audits and penetration
tests on its information systems. However, little progress has been made in securing
infrastructures from failures in other, interconnected infrastructures on which they depend to
work efficiently. Modern infrastructures are often depended on other infrastructures to function
properly. This necessity has led to the development of complex networks of interdepended
infrastructures. These dependency graphs of hide information about what will happen if a
failure occurs; in other words, they are as safe as their most critical path of interdependencies
and as exposed as their most dangerous node. In addition (and in a lower granularity level) high
level software applications nowadays are capable of controlling important machinery in critical
infrastructures from a distance. Even though it is common knowledge that software applications
are known to manifest flaws in their business logic implementation (either due to user or
programming errors), yet little to no effort has been made in finding a way to secure the
software assets that handle important machinery inside critical infrastructures from user error.
Logical flaws might lead to a serious critical infrastructure failure and even initiate cascading
effects between interconnected infrastructures.
x
In this context, the process of analyzing the security of a critical infrastructure appears to be
twofold: First and foremost, we must not only secure each infrastructure alone, but must also
be in a position to pinpoint potential cascading effects amongst depended critical infrastructures
and analyze what their impacts would be along with the likelihood of their occurring, same as
proposed for a single critical infrastructure in (DHS, 2013). At the same time, since software
has taken all the burden of controlling most parts of a CI, securing a critical infrastructure means
securing their information systems at a software level; i.e. to secure the application ecosystem
used to control a critical infrastructure by offering mechanisms that protect critical
infrastructures from software vulnerabilities yet to be dealt with.
This work approaches security in critical infrastructures by developing possible, practical
solutions for both aforementioned granularity levels of infrastructure protection. This work
differentiates from the mainstream security literature concerning the protection of critical
infrastructures, by (a) developing a combined multi-methodology able to pinpoint and assess
cascading failures amongst critical infrastructures, (b) by developing algorithms and strategies
that can be implemented for the detection of infrastructures that affect many other CI and
successful risk mitigation in interconnected infrastructures and (c) by developing a method for
the automatic detection of logical errors, race conditions and vulnerabilities in high-level
software used in CIs; specifically, in software able to remotely control machinery in critical
infrastructures. To the best of our knowledge, this is the first method able to detect logical errors
in diverse situations and diverse types of software. Although our initial target was to use the
aforementioned method in software handling machinery, the method can be used in a wider
basis to detect business logic errors in all types of software.
To raise the bar, this dissertation also presents two novel tools, PLATO and CIDA that are
manifestations of the proposed methods for securing a critical infrastructures in both analyzed
granularity levels. These tools perform automated analysis and do not require significant
technical expertise from users. Specifically, PLATO is a tool able to automatically detect some
types of logical errors in software. It, amongst other experiments, it was used to detect logical
errors in high-level capable of controlling infrastructure machinery remotely. Furthermore, a
macroscopic prediction model that identifies potential dangerous dependency chains between
critical infrastructures and the evolution of hazards over the course of time is proposed, coupled
with a graph theory analysis model able to pinpoint dangerous standalone Critical
infrastructures that greatly affect the overall stability of a graph of interconnected critical
infrastructures. These methods were implemented in CIDA, a tool that can model cascading
effect between infrastructures, analyze the evolution of hazards and, finally, pinpoint dangerous
Cis able to cause massive damage in multiple interconnected infrastructures; if a failure were
to manifest on them.
xi
As a case study, this work utilizes known real-world cascading failure scenarios and software
applications to test both developed tools. On the one hand, it explores whether – and under
which circumstances – logical errors that affect execution and cause failures can be detected in
source code (the PLATO too). On the other, it analyzes real-world cascading effect scenarios
like the California Blackout Scenario to test whether the interdependency analysis
methodologies can efficiently detect cascading failures and dangerous Cis.
This work provides evidence suggesting that more work is required for the protection of
critical infrastructures, while they use high-level software to control machinery and, at the same
time, are depended on other infrastructures also prone to failures.
This thesis interpolates material from three journal papers by the author and from various
conference papers (Stergiopoulos, 2012) (Stergiopoulos, 2013) (Stergiopoulos, 2014)
(Stergiopoulos, 2015). Chapter 3’s research core is based on (Vigna, 2011) and some of the
aforementioned publications coauthored with Ass. Prof. Panagiotis Katsaros and Prof. Dimitris
Gritzalis, respectively. Some material from each of these papers has also been incorporated into
this introductory Chapter. Meanwhile, Chapters 4 and 5 use material from (Kotzanikolaou, 2010)
(Kotzanikolaou, 2011) (Kotzanikolaou, 2012) (Kotzanikolaou, 2013) along with two of the
aforementioned journals (Stergiopoulos, 2015) (Stergiopoulos, 2015), coauthored with Ass. Prof.
Panagiotis Kotzanikolaou and Prof. Dimitris Gritzalis.
xii
(this page is intentionally left blank)
xiii
Extended abstract (in Greek)
Μια Κρίσιμη Υποδομή αποτελεί την ραχοκοκκαλιά της οικονομίας, της ασφάλειας και του
βιωτικού επίπεδου ενός έθνους. Είναι το είδος της υποδομής το οποίο παρέχει ενέργεια και νερό
στα σπίτια, υποστηρίζει τις μεταφορές και τα συστήματα επικοινωνίας στα οποία στηρίζεται η
κοινωνία. Τα κριτήρια που προσδιορίζουν πότε μια υποδομή θεωρείται χρήσιμη, έχουν
διευρυνθεί εν παρόδω χρόνω (DHS, 2013). Ως κρίσιμη υποδομή ορίζεται το σύνολο εκείνων των
αγαθών, των συστημάτων και των δικτύων, είτε φυσικών είτε εικονικών, τα οποία είναι τόσο
απαραίτητα για μια χώρα ώστε πιθανή μη-διαθεσιμότητά ή καταστροφή τους θα είχε τεράστιες
και εξουθενωτικές επιπτώσεις στην εθνική ασφάλεια, οικονομία, την δημόσια υγεία ή
οποιονδήποτε συνδυασμό αυτών (DHS, 2013).
Μια αυξανόμενη λίστα των υποδομών που χρήζουν προστασίας θα απαιτήσει από κάθε
ομοσπονδιακή κυβέρνηση να ιεραρχήσει τις προσπάθειές της και να προσπαθήσει να
ελαχιστοποιήσει τις επιπτώσεις από τυχόν μελλοντική αστοχία οποιουδήποτε είδους σε κρίσιμες
υποδομές της χώρας (π.χ. τρομοκρατική επίθεση ή αστοχία συστημάτων), ενώ, ταυτόχρονα,
λαμβάνοντας υπόψη το είδος αυτών των επιπτώσεων και την πιθανότητα εμφάνισης τους (Motef
et al., 2003).
Λαμβάνοντας υπόψη όλα τα ανωτέρω, είναι εμφανές πως οι κρίσιμες υποδομές μαζί με τις
υπηρεσίες και τα συστήματά τους, πρέπει να προστατεύονται επαρκώς ενάντια σε όλα τα είδη
αστοχιών ή ατυχημάτων, είτε πηγάζουν από ανθρώπινες πράξεις είτε από φυσικά φαινόμενα. Οι
κρίσιμες υποδομές παρέχουν υπηρεσίες αναγκαίες για ένα έθνος προκειμένου να λειτουργήσει
ορθά και να μπορεί να υποστηρίζει τις ανάγκες των πολιτών του, διαμέσου υπηρεσιών όπως το
σύστημα Υγείας, οι μεταφορές κτλ. Ακόμα περισσότερο αφού τέτοιου είδους αστοχίες μπορούν
να προξενηθούν από κακόβουλους επιτιθέμενους άλλων χωρών με σκοπο να πλήξουν το ίδιο το
έθνος και τα εισοδήματά του.
Μέχρι στιγμής, η δημοσιευμένη έρευνα έχει προσπαθεί να ασφαλίσει τις κρίσιμες υποδομές
μέσω διαδικασιών αξιολόγησης του ρίσκου και των απειλών που ενδεχομένως μπορούν να
πλήξουν μια κρίσιμη υποδομή. Μεθοδολογίες βασισμένες σε διεθνή πρότυπα ISO (όπως το ISO
27001) και αντίστοιχες αναλύσεις ασφάλειας, η οποίες προμηθεύουν τις κρίσιμες υποδομές με
σχέδια ασφάλειας και μέτρα προστασίας ικανά να θωρακίσουν τις υποδομές ενάντια σε πληθώρα
απειλών. Ωστόσο, ελάχιστη πρόοδος έχει γίνει όσων αφορά το θέμα της προστασίας των
υποδομών απέναντι σε αστοχίες που προέρχονται από το εξωτερικό της περιβάλλον,
συγκεκριμένα από άλλες υποδομές στις οποίες στηρίζεται μια υποδομή για να διασφαλίσει την
απρόσκοπτη λειτουργία της. Οι σύγχρονες κρίσιμες υποδομές συχνά εξαρτώνται από άλλες
υποδομές προκειμένου να παρέχουν τις υπηρεσίες τους ορθώς και δίχως διακοπές. Αυτή η
ανάγκη έχει οδηγήσει στην δημιουργία πολύπλοκων δικτύων από αλληλο-εξαρτώμενες
υποδομές. Αυτοί οι γράφοι αλληλο-εξαρτήσεων κρύβουν πληροφορία σχετικά με το τι θα
xiv
μπορούσε να συμβεί, αν μια αστοχία έβγαζε μια εκ των υποδομών εκτός λειτουργίας. Με άλλα
λόγια, αυτοί οι γράφοι κρίσιμων υποδομών είναι τόσο ασφαλείς όσο το πιο κρίσιμο μονοπάτι
αλληλοεξαρτήσεων που έχουν και είναι τόσο εκτεθειμένοι όσο ο πιο επικίνδυνος κόμβος-
υποδομή τους. Επίσης (και όσων αφορά ένα χαμηλότερο επίπεδο ανάλυσης της ασφάλειας
υποδομών), ελάχιστη πρόοδος έχει σημειωθεί και στην προστασία των υποδομών από το
ανθρώπινο λάθος, όταν υπαλληλοι χειρίζονται λογισμικό που ελέγχει μηχανήματα στα οποία
βασίζονται οι κρίσιμες υποδομές, ιδιαίτερα όταν αυτές οι λειτουργίες ελέγχονται
απομακρυσμένα. Παρόλο που αποτελεί κοινή γνώση το γεγονός πως το λογισμικό συχνά περιέχει
λογικά σφάλματα στην υλοποιημένη επιχειρησιακή του λογική, σφάλματα τα οποία εισάγονται
κατά την ανάπτυξή του, ωστόσο ελάχιστος χρόνος έρευνας έχει αφιερωθεί για την εύρεση ενός
τρόπου εντοπισμού αυτών των λαθών και διασφάλισης της απρόσκοπτης λειτουργίας του
λογισμικού, ιδιαίτερα λογισμικού που είναι υπεύθυνο σε μια κρίσιμη υποδομή για πληθώρα
υπηρεσιών. Τα λογικά σφάλματα μέσα σε τόσο σημαντικό λογισμικό μπορεί να οδηγήσουν σε
σημαντικότατες αστοχίες, ακόμα και να εκκινήσουν μια αλυσιδωτή αντίδραση αστοχιών μεταξύ
κρίσιμων υποδομών που διασυνδέονται.
Έχοντας αυτά υπόψη, η διαδικασία θωράκισης των κρίσιμων υποδομών από κινδύνους και
αστοχίες που ακόμα δεν έχουν διερευνηθεί καταλλήλως, διαφαίνεται πως είναι διμερής: Κατ'
αρχάς, θα πρέπει κάθε υποδομή να προστατευθεί όχι ως μια αυτοτελής οντότητα, αλλά
ταυτόχρονα και ως μέρος μιας αλυσίδας συνδεδεμένων κρίσιμων υποδομών. Οι υπεύθυνοι
ασφάλειας πρέπει να είναι σε θέση να εντοπίσουν τις πιθανές επιδράσεις αλυσιδωτών αστοχιών
μεταξύ αλληλο-εξαρτημένων κρίσιμων υποδομών και να αναλύσουν τις επιπτώσεις τους και την
πιθανότητα εμφάνισης τους, ακριβώς όπως επιχειρείται να αναλυθεί μια κρίσιμη υποδομή κατά
μόνας (DHS, 2013). Την ίδια στιγμή, ασφαλής κρίσιμη υποδομή θεωρείται κάθε υποδομή που
προστατεύει τα πληροφοριακά της συστήματα και σε επίπεδο λογισμικού, με άλλα λόγια η
προστασία το οικοσυστήματος προγραμμάτων που ελέγχουν μια κρίσιμη υποδομή μέσω
μηχανισμών για τον έλεγχο, την προστασία και την διαχείριση λογισμικού από ευπάθειες που
ίσως να μην έχουν εντοπιστεί.
Η παρούσα διπλωματική προσεγγίζει το θέμα της ασφάλειας των κρίσιμων υποδομών μέσω
της ανάπτυξης πρακτικών λύσεων για την αντιμετώπιση και των δύο προαναφερθέντων
κατηγοριών, σε όλα τα επίπεδα ανάλυσης. Η παρούσα δουλειά διαφοροποιείται από την
πεπατημένη βιβλιογραφία ασφάλειας που αφορά την προστασία των κρίσιμων υποδομών με τους
εξής τρόπους:
a) Αναπτύσσει μια συνδυαστική πολυ-μεθοδολογία ικανή να εντοπίζει και να αξιολογεί
διαδοχικές αστοχίες μεταξύ αλληλο-εξαρτώμενων κρίσιμων υποδομών,
xv
b) Αναπτύσσει αλγορίθμους και στρατηγικές που μπορούν να εφαρμοστούν πρακτικά
προκειμένου να μετριάσουν το ρίσκο πιθανών διαδοχικών αστοχιών ανάμεσα σε
αλληλο-εξαρτώμενες υποδομές,
c) Αναπτύσσει μια μεθοδολογία για τον αυτόματο εντοπισμό λογικών λαθών, συνθηκών
ανταγωνισμού και ευπαθειών που προέρχονται από λανθασμένη υλοποίηση
επιχειρησιακής λογικής μέσα στο λογισμικό και ενδεχομένως να μπορούν να
οδηγήσουν σε σοβαρότατες αστοχίες λογισμικού. Συγκεκριμένα, σε λογισμικό που
μπορεί να χειρίζεται μηχανήματα κρίσιμων υποδομών απομακρυσμένα. Απ’ όσο
γνωρίζουμε, πρόκειται για την πρώτη υλοποιημένη μέθοδο που είναι ικανή να
εντοπίζει λογικά σφάλματα που μπορούν να οδηγήσουν σε αστοχίες υλικού λόγω
λανθασμένης υλοποίησης της λειτουργικότητας. Και να μπορεί να το κάνει αυτό σε
διαφορετικά είδη λογισμικού. Παρόλο που ο πρωτεύων στόχος ήταν να
χρησιμοποιηθεί η παρούσα μεθοδολογία για την προστασία λογισμικού στις κρίσιμες
υποδομές, η μέθοδος αποδείχτηκε ότι μπορεί να χρησιμοποιηθεί σε μια πολύ ευρύτερη
βάση και σε μια πληθώρα εφαρμογών.
Προκειμένου να ανεβάσουμε ακόμα περισσότερο τον πήχη, η παρούσα διατριβή παρουσιάζει
επίσης δύο νέα εργαλεία, το PLATO και το CIDA που πρόκειται για υλοποιήσεις των
προαναφερθέντων προτεινόμενων μεθόδων για την προστασία των υποδομών ζωτικής σημασίας.
Τα εργαλεία αυτά εκτελούν αυτοματοποιημένη ανάλυση και δεν απαιτούν σημαντική
τεχνογνωσία από τους χρήστες τους. Συγκεκριμένα, το PLATO είναι ένα εργαλείο που μπορεί
να ανιχνεύσει αυτόματα ορισμένους τύπους λογικών σφαλμάτων λογισμικού. Μεταξύ άλλων
πειραμάτων, χρησιμοποιήθηκε για να ανιχνεύσει λογικά σφάλματα σε εφαρμογές ικανές να
ελέγχουν εξ αποστάσεως μηχανήματα σε κρίσιμες υποδομές. Επιπλέον, προτείνεται ένα
πανοπτικό μοντέλο πρόβλεψης ικανό να εντοπίζει εν δυνάμει επικίνδυνες αλυσίδες αλληλο-
εξαρτώμενων κρίσιμων υποδομών, αλλά και να προβλέπει την εξέλιξη των αστοχιών μέσα σε
αυτές εν παρόδω χρόνω. Το μοντέλο αυτό συμπληρώνεται με μία μεθοδολογία ανάλυσης γράφων
ώστε να είναι σε θέση να εντοπίσει επικίνδυνες κρίσιμες υποδομές μέσα σε ένα δίκτυο αλληλο-
εξαρτώμενων υποδομών, οι οποίες ενδεχομένως να επηρεάζουν σημαντικά τη συνολική
σταθερότητα του συστήματος. Αυτό το πολυ-μοντέλο υλοποιήθηκε μέσα σε ένα εργαλείο με το
όνομα CIDA, ένα εργαλείο που μπορεί να μοντελοποιήσει την καταλυτική δράση των αστοχιών
μεταξύ των υποδομών, να αναλύσει την εξέλιξη των αστοχιών και τις επιπτώσεις αυτών και,
τέλος, να εντοπίσει κρίσιμες υποδομές που είναι σε θέση, σε περίπτωση κατάρρευσής τους, να
προκαλέσουν μαζική καταστροφή σε πολλές διασυνδεδεμένες υποδομές ταυτόχρονα.
Τα κεφάλαια της παρούσας διατριβής συνοψίζονται ως εξής:
Το Κεφάλαιο 1 περιλαμβάνει την αφορμή της έρευνας, το σκοπό και τη μέθοδο εργασίας μας
καθώς και τη συνοπτική περιγραφή της ερευνητικής συνεισφοράς στο γνωστικό πεδίο.
xvi
Στο Κεφάλαιο 2 παρουσιάζεται το υπόβαθρο της διατριβής. Παρέχεται ο ορισμός των
κρίσιμων υποδομών, του λογισμικού, των ευπαθειών και των αστοχιών που υιοθετείται στα
πλαίσια της διατριβής. Έπειτα, παρατίθενται οι διαδικασίες ανάλυσης του κινδύνου στις κρίσιμες
υποδομές, τα μεθοδολογίες ανάλυσης ασφάλειας των κρίσιμων υποδομών όσο και του
λογισμικού αυτών. Το κεφάλαιο, επίσης, αναφέρει τις προκλήσεις που εισάγουν οι κρίσιμες
υποδομές στη διασφάλιση της ασφάλειας των πολιτών ενός έθνους.
Στο Κεφάλαιο 3 μελετάται ο αυτόματος εντοπισμός λογικών λαθών. Συγκεκριμένα,
προτείνεται μια μέθοδος, η οποία υλοποιείται και σε ένα εργαλείο με το όνομα PLATO, η οποία
μπορεί να εντοπίσει διάφορα είδη λογικών λαθών σε διάφορα είδη εφαρμογών, από απλές
εφαρμογές μέχρι εφαρμογές ικανές να ελέγχουν απομακρυσμένα μηχανήματα κρίσιμων
υποδομών. Εξετάζονται τρόποι ανάλυσης της λογικής του λογισμικού υπό την μορφή
μαθηματικών κατηγορημάτων και η χρήση αυτών για εντοπισμό λαθών που αλλιώς περνούν
απαρατήρητα από τεχνικές αναλύσεις και μελέτες πηγαίου κώδικα. Στο κεφάλαιο μελετάται
επίσης ο τρόπος εντοπισμός και η αξιολόγηση τέτοιων πιθανών σφαλμάτων προκειμένου να
προβλεφθούν πιθανά σφάλματα που μπορούν να οδηγήσουν σε σοβαρές αστοχίες λογισμικού και
υλικού.
Το Κεφάλαιο 4 περιλαμβάνει μεθοδολογίες ανάλυσης αλληλο-εξαρτήσεων μεταξύ κρίσιμων
υποδομών προκειμένου να προτείνει αποδοτικές τεχνικές μείωσης του ρίσκου και του κινδύνου
εμφάνισης αστοχιών στις κρίσιμες υποδομές. Εξερευνά τις συσχετίσεις και τις συνδέσεις
μεταξύ μονοπατιών που δημιουργούνται από αλληλο-εξαρτώμενες υποδομές, κάνοντας χρήση
της θεωρίας γράφων. Παρουσιάζεται πώς οι μετρικές θεωρίας γράφων μπορούν να
εφαρμοστούν για τον εντοπισμό και την αξιολόγηση εναλλακτικών στρατηγικών απομείωσης
ρίσκου μεταξύ των υποδομών. Τα ερευνητικά πειράματα που παρουσιάζονται βασίζονται πάνω
σε τυχαίους γράφους αλληλο-εξαρτώμενων υποδομών που προσομοιώνουν υπάρχουσες
κρίσιμες υποδομές και διασυνδέσεις αυτών, όπως αυτά δημοσιεύτηκαν σε πρόσφατες μελέτες.
Τέλος, χρησιμοποιώντας τα αποτελέσματα των προαναφερθέντων πειραμάτων, προτείνεται
ένας αλγόριθμος ιεράρχησης της σημαντικότητας κρίσιμων υποδομών μέσα σε ένα γράφο
αλληλο-εξαρτημένων υποδομών προκειμένου να εξεταστούν συγκεκριμένα μέτρα προστασίας
για μεγιστοποίηση της μείωσης του κινδύνου σε όλο το σύστημα εξαρτημένων υποδομών.
Στο Κεφάλαιο 5, μελετάται και επεκτείνεται μια μέθοδος ανάλυσης αλληλο-εξαρτήσεων
κρίσιμων υποδομών, προκειμένου να επιτευχθεί μια δυναμική αξιολόγηση διαδοχικών αστοχιών
μεταξύ υποδομών εν παρόδω χρόνω. Προτείνεται η χρήση διαφορετικών μαθηματικών μοντέλων
ανάπτυξης προκειμένου να αποδοθούν επαρκώς τα φαινόμενα αργής, γραμμικής και εκθετικής
εξέλιξης των αστοχιών στις κρίσιμες υποδομές, αλλά αντί να γίνεται χρήση απλών, στατικών
προβολών του κινδύνου, προτείνεται η «αντικειμενική» προσέγγιση της ασάφειας των
επιπτώσεων στις αστοχίες μέσω ενός εξελιγμένου μοντέλου ασαφούς λογικής το οποίο λαμβάνει
xvii
υπόψη του και τις γειτονικές εξαρτήσεις υποδομών και πιθανές αστοχίες αυτών. Τέλος,
παρουσιάζεται η υλοποίηση της ανωτέρω μεθοδολογίας στο εργαλείο CIDA και παρουσιάζονται
τα αποτελέσματα από την χρήση του εργαλείου σε ένα αληθινό σενάριο καταστροφής το οποίο
συνέβη στην Καλιφόρνια. Το κεφάλαιο κλείνει με την παρουσίαση των επιδόσεων του εργαλείου
στην πρόβλεψη των αστοχιών και των επιπτώσεων αυτής.
Η διατριβή ολοκληρώνεται στο Κεφάλαιο 6 με τη σύνοψη και προτάσεις για μελλοντική
έρευνα.
xviii
Table of Contents
List of Figures .......................................................................................................... xxii
List of Tables ........................................................................................................... xxii
List of Acronyms ..................................................................................................... xxii
Chapter 1: Introduction ....................................................................................... 1
1.1 Research motivation ........................................................................................................ 1
1.1.1 Motivation for analyzing infrastructure interdependencies ...................................... 1
1.1.2 Motivation for analyzing high-level software ........................................................... 3
1.2 Research statement and approach .................................................................................... 4
1.3 Contributions ................................................................................................................... 5
1.3.1 Contributions in logical error detection and software system failures...................... 5
1.3.2 Contributions in infrastructure dependency analysis along with mitigation of
cascading and common-cause failures ............................................................................... 6
1.4 Dissertation outline ......................................................................................................... 8
Chapter 2: Background ...................................................................................... 11
2.1 Definitions of Critical Infrastructure ............................................................................. 11
2.2 Risk Analysis in all granularity levels ........................................................................... 13
A. Establish the Scope .................................................................................................. 13
B. Identify the Risks ..................................................................................................... 14
C. Analyze the Risks .................................................................................................... 15
D. Treat the Risks ......................................................................................................... 15
2.4 Threat model ................................................................................................................. 16
2.5 Summary ....................................................................................................................... 17
Chapter 3: Automatically detecting logical errors, race conditions and IT
infrastructure failures in software using dynamic invariant assertions and
symbolic execution .................................................................................................... 20
3.1 Introduction ................................................................................................................... 20
3.2 Related Work ................................................................................................................. 22
3.3 Analysis Building Blocks .............................................................................................. 23
3.3.1. Dynamic Invariants for profiling the behavior of the source code ........................ 23
3.3.2. Symbolic execution for verifying dynamic invariants ........................................... 24
3.3.3. Fuzzy Logic classification of detections ................................................................ 25
3.3.3.1. Severity ........................................................................................................... 26
3.3.3.2. Measuring Severity of execution paths using its Statistical Information
Gain ............................................................................................................................. 27
3.3.3.3. Reliability ....................................................................................................... 29
3.3.3.5. Risk: Combining Severity and Reliability ratings .......................................... 31
3.4 A method to detect logical errors in source code .......................................................... 32
3.4.1. The method’s workflow ......................................................................................... 33
xix
3.4.2. PLATO’s architecture ............................................................................................ 33
3.4.3. Classifying execution paths ................................................................................... 35
3.4.3.1 A Taxonomony of instructions for Severity calculations ............................ 35
3.4.3.2. Instructions for Reliability calculations .......................................................... 37
3.4.4. Symbolic execution and PLATO’s JPF listener .................................................... 37
3.5 Experimental results ................................................................................................ 38
3.5.1 An example AUT .................................................................................................... 38
3.5.2 Experiment 1: real-world airline test from the SIR Object Database .................. 39
3.5.3 Experiment 2: Manipulating remote SCADA RTUs (logical error CWE 840) ... 41
3.5.4 Experiment 3: Real-world injection test - NASA’s Apollo Lunar Lander 790
controller .......................................................................................................................... 45
3.6 Discussion and comparisons .................................................................................... 47
3.7 Advantages and Limitations .................................................................................... 48
3.8 Summary ....................................................................................................................... 48
Chapter 4: Risk Mitigation Strategies for Critical Infrastructures Based on
Graph Centrality Analysis ....................................................................................... 51
4.1 Introduction ................................................................................................................... 51
4.2 Building Blocks ............................................................................................................ 53
4.2.1. Multi-risk dependency analysis methodology ....................................................... 53
4.2.1.1. First-order dependency risk ............................................................................ 53
4.2.1.2. Extending to n-order dependency risk ............................................................ 53
4.2.2. Graph centrality metrics ......................................................................................... 54
4.2.3. Feature selection .................................................................................................. 54
4.3 Exploring Centrality Metrics in Dependency Risk Graphs ........................................... 55
4.3.1. Analysis of centrality metrics ................................................................................ 56
4.3.1.1. Degree centrality ............................................................................................. 56
4.3.1.2. Closeness centrality ........................................................................................ 56
4.3.1.3. Eccentricity centrality ..................................................................................... 57
4.3.1.4. Betweenness centrality ................................................................................ 57
4.3.1.5. Bonacich (Eigenvector) centrality .................................................................. 57
4.3.2. Applying feature selection to classify centrality metrics ....................................... 58
4.3.2.1. Results of feature selection for centrality metrics ........................................... 59
4.4 An Algorithm for Efficient Risk Mitigation Strategy.............................................. 60
4.4.1. Modeling feature selection results to algorithmic restrictions ............................... 60
4.4.1.1. Model Definition ............................................................................................ 61
4.4.1.2. Setup ............................................................................................................... 62
4.4.1.3. Using decision trees and Information Gain for node selection ....................... 62
4.4.1.4. Example: Traversing a decision tree using Information Gain splits ............... 63
4.5 Validation of the proposed algorithm ...................................................................... 64
xx
4.5.1. Efficiency analysis of the proposed mitigation strategy .................................. 65
4.5.1.1. Effect of the algorithm in the most critical path ............................................. 66
4.5.1.2. Effect of the algorithm in the top 20 risk paths .............................................. 66
4.5.1.3. Effect of the algorithm in the entire dependency risk graph ........................... 68
4.5.2. Comparing results with alternative mitigation strategies ....................................... 68
4.5.2.1. Using initiators from the most critical paths ................................................... 68
4.5.2.2. Using nodes with high Degree Centrality (sinkholes and outbound) ............. 69
4.5.2.3. Comparison results ......................................................................................... 69
4.5.3. Implementation details ........................................................................................... 69
4.6. Conclusions .................................................................................................................. 70
4.6.1. Advantages and Limitations of the proposed mitigation method .......................... 70
Chapter 5: Time-based Critical Infrastructure Dependency Analysis for
Large-Scale and Cross-Sectoral Failures................................................................ 72
5.1 Introduction ................................................................................................................... 72
5.1.1. Large-scale and cross-sectoral dependencies ......................................................... 73
5.2. Building Blocks ............................................................................................................ 73
5.2.1. Multi-risk dependency analysis methodology ....................................................... 73
5.2.1.1. First-order dependency risk ............................................................................ 74
5.2.1.2. Risk of n-order dependencies ......................................................................... 75
5.2.2 Fuzzy Logic ............................................................................................................ 76
5.3. A Method for Time-Based Analysis of Cascading and Common Cause Failures ........ 77
5.3.1. Modeling Time Analysis of Cascading Failures .................................................... 77
5.3.1.1. Model Definition ............................................................................................ 77
5.3.1.2. Setup ............................................................................................................... 79
5.3.1.3. Calculating Fuzzy Time-based Impact Values ............................................... 81
5.3.1.4. Time-related Multi-order Dependency Risk ................................................... 82
5.3.2. Combining cascading and common-cause failure risks ......................................... 83
5.4. Design and Implementation of the CIDA tool .............................................................. 84
5.4.1. The Neo4J graph database ..................................................................................... 84
5.4.2. Design of the Dependency Graph Analysis Tool................................................... 85
5.4.3. Modeling CIs & Dependencies .............................................................................. 86
5.5. Efficiency analysis........................................................................................................ 87
5.6. Analyzing Real-World Scenarios of Cascading Effects ............................................... 89
5.6.1. Case study: A real-world cascading blackout failure............................................. 89
5.6.2. A cascading-only dependency failure scenario ...................................................... 92
5.6.3. Combined common-cause and cascading scenarios .............................................. 94
5.7. Discussion and Comparison with Other Approaches .............................................. 94
5.7.1. Comparison with related methodologies and tools ................................................ 94
5.7.2. Advantages and Limitations .................................................................................. 97
xxi
Chapter 6: Conclusions....................................................................................... 99
6.1 Summary and discussion ............................................................................................... 99
6.2 Publications ................................................................................................................. 101
6.3 Future work ................................................................................................................. 103
6.4 Concluding remarks .................................................................................................... 104
Appendix .................................................................................................................. 107
A. Time-Based Analysis of cascading failures using the CIDA tool ................................ 107
A1. Tool libraries and development............................................................................... 107
A2. Fuzzy Logic IF THEN rules.................................................................................... 108
A3. Impact Time Tables ................................................................................................ 109
A3. Guide to installing the CIDA Eclipse project ......................................................... 117
References ................................................................................................................ 118
xxii
List of Figures Figure 1: ............................................................................................................................................. 32
Figure 2: ................................................................................................................................... .......... 34
Figure 3: ............................................................................................................................................. 37
Figure 4: ............................................................................................................................................. 38
Figure 5: ................................................................................................................................... .......... 39
Figure 6: ............................................................................................................................................. 40
Figure 7: ............................................................................................................................................. 40
Figure 8: ............................................................................................................................................. 40
Figure 9: ............................................................................................................................................. 42
Figure 10: ........................................................................................................................................... 43
Figure 11: ........................................................................................................................................... 43
Figure 12: ........................................................................................................................................... 46
Figure 13: ........................................................................................................................................... 66
Figure 14: ........................................................................................................................................... 67
Figure 15: ........................................................................................................................................... 68
Figure 16: ........................................................................................................................................... 78
Figure 17: ........................................................................................................................................... 87
Figure 18: ........................................................................................................................................... 89
Figure 19: ........................................................................................................................................... 90
Figure 20: ........................................................................................................................................... 91
Figure 21: ........................................................................................................................................... 92
Figure 22: ........................................................................................................................................... 93
List of Tables Table 1: ........................................................................................................................................... 11
Table 2: ........................................................................................................................................... 29
Table 3: ........................................................................................................................................... 31
Table 4: ........................................................................................................................................... 32
Table 5: ........................................................................................................................................... 36
Table 6: ........................................................................................................................................... 37
Table 7: ........................................................................................................................................... 45
Table 8: ........................................................................................................................................... 59
Table 9: ........................................................................................................................................... 59
Table 10: ......................................................................................................................................... 69
Table 11: ......................................................................................................................................... 75
Table 12: ......................................................................................................................................... 81
Table 13: ......................................................................................................................................... 91
List of Acronyms
CI Critical Infrastructure
AUT Application Under Test
xxiii
SUT Software Under Test
RTU Remote Terminal Unit
1
Chapter 1: Introduction
1.1 Research motivation
Most Critical Infrastructures (CIs) can be modeled as cyber-physical systems whose cyber
components control their underlying physical components. CIs are inherently complex systems
since they integrate heterogeneous platforms, proprietary systems, protocols and open
communication networks. In addition, Cis are usually interconnected and interdependent with
other CIs which may belong to different sectors (such as energy, ICT or transportation).
According to (Rinaldi, 2001) CIs may have physical, informational or logical dependencies
between them. Thus, a failure1 in one infrastructure may affect the operation of other Cis due
to their dependencies. In the case of geographical dependency, seemingly independent CIs may
be affected by a threat due to their physical proximity. The protection from such types of
dependency failures is an active and recent area of research, as shown by the numerous projects
on the topic, e.g. DIESIS (Usov, 2010) (Rome, 2009), I2Sim (Marti, 2008), CIPRNet (Critical
infrastructure resilience network, 2014). Dependency modeling, simulation and analysis
(MS&A) has been studied extensively, using various approaches.
Disruptions or outages in CIs are usually categorized as cascading, escalating, or common-
cause (Rinaldi, 2001).
• A cascading failure is defined as a failure in which a disruption in an infrastructure A
affects one or more components in another infrastructure, say B, which in turn leads to the
partial or total unavailability of B.
• An escalating failure is defined as a failure in which disruption in one infrastructure
exacerbates an independent disruption of another infrastructure, usually in the form of
increasing the severity or the time needed for recovering from the second failure.
• A common-cause failure occurs when two or more infrastructure networks are disrupted at
the same time: components within each network fail because of some common cause. This
occurs when two infrastructures are co-located (geographic interdependency) or if the root
cause of the failure is widespread (e.g. a natural or a man-made disaster).
1.1.1 Motivation for analyzing infrastructure interdependencies
Recently, several dependency analysis methodologies and tools have been proposed,
focusing either on the impact (Franchina, 2011), consequences (Robert, 2004) or on the risk
derived from CI dependencies (Utne, 2011) (Kjølle,2012) (Kotzanikolaou, 2011)
(Kotzanikolaou, 2013) (Kotzanikolaou, 2013) and their potential cascading effect.
2
Existing methodologies and tools are usually sector-specific, oriented to power distribution
(e.g. (Carreras, 2012)) or to water distribution networks (e.g. (Judi, 2007)). These tools are very
useful for low-level analysis of small-scale scenarios; for example to identify the critical
components within a power transmission network. However, they may fall short when high-
level analysis is needed in order to model large-scale, cross-sectoral scenarios. For example,
the identification of dependency paths of high economic or societal risk that affect different
sectors. In practice, the CI operators perform risk assessments at an organization-level and may
not have knowledge (or interest) on threats coming from other dependent CIs. Such knowledge
can be acquired, to some extent, by taking part in international table-top exercises (Luiijf,
2010). Even though the economic impact of a failure can be assessed by a CI operator
(organizational-wise), the overall impact (or risk) of a given CI failure on other dependent CIs
is not a tangible number, especially when dealing with multi-order dependencies.
The need for a high-level multi-sectoral risk assessment has been recognized by international
bodies and policies (The post 2015 Hyogo Framework for Action, 2015). A high-level risk
analysis allows the identification of complex cascade or common-cause risk paths and the
comparison of alternative mitigation strategies. Note that a multi-layer risk assessment is not
an alternative to organization-wide risk assessments, since these are prerequisite input for a
multi-risk analysis. Such analysis would require modeling and analyzing hundreds or even
thousands of CIs. If a time-based analysis is to be performed, then complexity is even higher.
Unfortunately the computation of the cumulative security risks and the identification of the
critical points of failure is NP-complete and thus suboptimal, yet usable dependency analysis
tools have to be developed.
Another issue is that existing risk mitigation methodologies are usually empirical, and they
focus on CIs which are possible initiators of cascading failure and usually belong to specific
sectors such as Energy and ICT. By studying actual large-scale failures, it is obvious that these
two sectors often initiate serious cascading effects amongst interdependent CIs (e.g. the famous
California blackout scenario or the recent New York blackout scenario of 2015). However, by
only focusing on some sectors and on potential initiators may not be suitable in every case. The
impact of each dependency should be taken into account together with the position of each CI
within a network of interdependent CIs.
The systematic identification of the most important nodes and the prioritization of nodes for
the application of security controls can therefore be a complex task. The need for a high-level
and efficient risk mitigation technique that takes into account multiple characteristics of the
interdependent CIs is a well-known fact (Critical infrastructure resilience network ,2014). An
optimal risk mitigation methodology would allow the detection of the smallest subset of CIs
Introduction
3
that have the highest effect on the overall risk reduction on a risk graph, even though the
candidate nodes might not initiate a critical path or may not even belong to the most critical
paths.
1.1.2 Motivation for analyzing high-level software
As already stated, securing critical infrastructures is a complicated problem requiring
multiple approaches in different granularity levels. This thesis tries to approach infrastructure
security from yet another approach: Securing high-level software that controls infrastructure
machinery from flaws and errors in its logic that can lead to critical failures in systems.
When industries started connecting critical infrastructure systems (like SCADA systems)
with the Internet, these control systems were exposed to various vulnerabilities and threats
(Krutz, 2005). Built as standalone, isolated control systems, they lacked the proper security
measures needed to support a robust and safe functionality over the Internet. For example, an
over-the-Internet, man-in-the-middle attack on a green diesel generator that led to a total
meltdown has been presented in the literature (Alcaraz, 2011).
Obviously, the consequences of maliciously operating SCADA systems can be devastating.
Modern implementations are using the internet and allow common programming languages
(such as Java) to send and receive data and commands through the central control unit to RTUs.
Such implementations include, amongst others, openSCADA (Openscada, 2014), MODBUS4J
(Krutz, 2005) (Chikuni, 2007) and JAMOD (Wimberger, 2004), which uses the MODBUS
protocol. The added complexity coupled with open-source components resulted in an increase
in software weaknesses, vulnerabilities and failures in these types of systems (Cardenas, 2008).
A number of logical threats have been registered in public databases (Chikuni, 2007). Source
code analysis of programs that remotely control RTUs opens a path to prevent software failures.
Such failures may lead to malfunction in individual controllers or infrastructures, or even
initiate cascading failure to dependent infrastructures. Yet, to our knowledge, no effort has been
made in detecting logical flaws in the functionality of software that controls RTUs; flaws due
to erroneous implementation of the intended functionality in source code. Research on
automated detection of software errors and vulnerabilities has mainly focused on static analysis
and software model checking techniques that are effective in detecting a priori specified errors
(e.g. Time Of Check - Time Of Use errors, null pointer dereferences etc.), bad coding patterns
and some types of exploitable vulnerabilities (e.g. unsanitized input data, buffer overflows etc.)
(Codepro 2015) (Ucdetector, 2015) (Pmd, 2015) (Hovemeyer, 2004) (Coverity, 2015)
(Pasareanu, 2004) (Java pathfinder tool, 2015). Yet, errors related to the intended program
functionality, which are broadly called logical errors, are not a priori known. In a code auditing
process, they cannot be analyzed as pattern-specific errors since they are rather application-
specific. At the level of the program’s execution flow, these errors will cause execution
4
diversions that manifest as unintended program behavior (Felmetsger, 2010). A typical example
is the following:
Example 1. Consider an RTU that controls the gas supply in an infrastructure, using coils; a
faulty implementation of business logic in software can lead the program execution to bypass
some pressure checks and allow sending the same “increase flow” command more times than
what the pipe system can handle, eventually making pressure reach critical levels. Automated
detection of such software behavior is a relatively uncharted territory.
Modern automated source code analysis techniques can be very successful in detecting a
priori defined defect patterns and security vulnerabilities. Yet, they cannot detect flaws that
manifest due to erroneous translation of the software’s functional requirements into the source
code. The automated detection of logical errors that are attributed to a faulty implementation of
applications’ functionality is a relatively uncharted territory. We propose a combination of
automated analyses for logical error detection. As a proof of concept, our method has been
implemented in a prototype tool called PLATO that can detect various types of logical errors.
1.2 Research statement and approach
Under the aforementioned circumstances, it is evident that critical infrastructures are
currently vulnerable and must be protected against failures originating both from
interdependencies with other CIs and from flawed critical, high-level software able to
manipulate their systems and machinery. More specifically, current risk mitigation approaches
deem adequate to analyze a critical infrastructure as a single entity rather as a node in a
complicated dependency graph. Also, they focus solely on security countermeasures against
technical and organizational issues and not on the software itself and its potential uses.
This is evident in many risk assessments. The measures proposed focus solely on one critical
infrastructure and its information system(s), while not taking into consideration neither CI
dependencies nor their custom software which sometimes is specifically developed to handle
an infrastructure’s machinery. Therefore, proposed countermeasures can only help mitigate risk
from specific types of attacks and failures. Our findings suggest that regardless of how strict a
risk assessment of a CI may be or how up-to-date its software is kept; it still leaves room for
many types of failures to manifest, originating from threats outside the CI itself whether it is a
different infrastructure or a software application.
In this thesis, steps are taken in order to (a) understand the interconnections between
infrastructures and the graphs they form (sometimes even across boarders in multiple nations)
and (b) understand logical flaws in software that can lead to system or machinery failures.
Moreover, it is noticed that the mainstream CI dependency analysis tools focus mainly on the
Introduction
5
‘nuts and bolts’ of both CI interdependency analysis while ignoring both the evolution of impact
from failures over time and the position of a CI inside the dependency risk graph where multiple
CI lie. On the lower granularity level, it is noticed that current state-of-the-art source code
analysis tools ignore user practices or the business logic implemented inside the source code
and rather focus on technical vulnerabilities and a priori known flaws (such as TOUCTOU or
buffer overflows). In this context, this thesis makes the following research statement:
The current state of critical infrastructure security fails to protect critical infrastructures
across nations, both from interdependencies with other infrastructures and from
programming errors inside their own critical software.
This thesis contains evidence that support this statement. Our findings suggest that is indeed
possible to mitigate risk in interconnected critical infrastructures both by analyzing their
dependencies inside a CI graph and by securing critical infrastructure software from logical
errors.
1.3 Contributions
This thesis makes the following contributions in two different areas that concern the security
of critical infrastructures.
1.3.1 Contributions in logical error detection and software system failures
The first area is that of detecting logical errors in software which may lead to system failures.
In this subject, this thesis made the following contributions:
We show how most types of information flow dependent logical errors can be detected
by classifying invariant violations and their corresponding execution paths. Detections are
classified in two different groups of sets as follows:
the Severity sets, quantifying the danger level of an execution path π (the
impact of an error, if it were to manifest on path π during execution). Severity is based
on an algorithm which uses Information Gain for classification from data mining.
the Reliability sets, quantifying a path π with an invariant violation based on
the size and complexity of the code traversed by path π.
We present PLATO, a prototype tool for the detection of logical errors, race conditions
and vulnerabilities in software applications able to perform various critical tasks. To the best
of our knowledge, PLATO is the first tool that can detect logical errors in diverse situations.
6
To test the range of errors that can be detected, we evaluate PLATO on three different
AUTs with different types of logical errors: (i) A multi-threaded airline control system for
selling airplane tickets, which is a real-world software that has been previously used in a
controlled experimentation with program analysis. (ii) A real-world software based on the
National Institute of Standards and Technology (Boland, 2012) that was intentionally designed
with logical errors leading to software vulnerabilities. This NIST-based software can handle
field sensors in a SCADA system and (iii) A controller from NASA’s Lunar Lander, where we
injected logically malformed data based on code metrics.
These diverse tests were chosen to show the flexibility of the proposed approach in detecting
logical errors in various situations. The fact that PLATO flagged all types of logical errors in
all tests serves as a proof of concept that our method is a viable solution for the automated
detection of logical errors.
1.3.2 Contributions in infrastructure dependency analysis along with mitigation of
cascading and common-cause failures
The second research area is that of analyzing the dependencies between interconnected
critical infrastructures, analyzing infrastructure graphs, pinpoint critical risk paths for cascading
failures along with dangerous CIs that greatly affect the entire web of interconnected
infrastructures and proposing mitigation strategies to lower the overall risk of failures in
interdepended CIs. In this part of our research endeavors, this thesis made the following
contributions:
We extend our recent work on CI dependency analysis (Kotzanikolaou, 2011)
(Kotzanikolaou, 2013) (Kotzanikolaou, 2013). We design and implement a Critical
Infrastructure Dependency Analysis tool (CIDA) (Stergiopoulos, 2014), which is based on risk
analysis and graph modelling. Our extensions are twofold:
a) We introduce time-based analysis models to study the evolution of dependency chains
during slow, linear or fast evolving cascading failures. Note that each dependency
may follow a different time model which is “fine-tuned” for each examined dependency
using fuzzy modelling.
b) We model concurrent cascading and common-cause failures, in order to effectively
analyze major failures.
In order to validate the applicability and efficiency of our approach, we stress-test our tool,
using random graphs of up to a thousand nodes (CIs) with randomly selected dependencies.
Our tests demonstrate the computational efficiency of CIDA for large-scale scenarios, under
reasonable parameters (i.e. maximum number of dependencies per node and maximum order
Introduction
7
of dependencies). As a proof of concept, we also run targeted tests based on data of real
cascading and common-cause failures.
CIDA is a proactive modelling and security dependency analysis tool for the analysis of
large-scale and cross-sectoral dependency scenarios. It allows risk assessors and CIP decision
makers to analyze complex dependency graphs and identify critical dependency chains before
an actual threat has occurred. Thus it can reveal underestimated dependency risks that need
further attention. Moreover, CIDA may also be used as an efficient tool for the assessment of
alternative risk mitigation strategies and therefore help increase CI resilience1.
Towards this direction, in this thesis we also explore the use of graph centrality metrics in
Dependency Risk Graphs, in order to effectively prioritize CIs when applying risk mitigation
controls. Our goal is to design, implement and evaluate the effectiveness of alternative risk
mitigation strategies. Each examined mitigation strategy is implemented algorithmically and it
is empirically evaluated through simulations. The ultimate goal is to identify the minimum
subset of CI nodes within a graph, whose risk treatment (application of security controls) would
result in the maximum risk reduction in the entire graph of interdependent CIs.
On pinpointing dangerous CIs in Dependency Risk Graphs and mitigating the risk failure,
this thesis extensions are threefold:
a) We run various data mining experiments to identify correlations between centrality metrics
and CI nodes that have high impact in a dependency risk graph;
b) We use the optimum centrality metrics, as identified in the previous step, in order to
develop and test various risk mitigation strategies that maximize risk reduction. Results
show that each mitigation strategy is most effective in achieving specific goals;
c) We take into consideration nodes with a high number of inbound (sinkholes) and outbound
connections and compare risk reduction results.
In order to validate the proposed mitigation strategies, we run experiments on hundreds of
random graphs with randomly selected dependencies. These tests demonstrate the efficiency of
the proposed risk mitigation strategies. All random graphs match criteria found in real-world
CI graphs of interconnected infrastructures.
The proposed mitigation strategies can be used as a proactive tool for the analysis of large-
scale dependency infrastructure interconnections and pinpoint underestimated infrastructures
that are critical to the overall risk of a dependency graph.
1 Resilience implies the ability to withstand accidental or deliberate threats or incidents
(Presidential Policy Directive, 2013). A resilience-oriented approach accepts that failures will occur;
thus CIs should implement controls that effectively absorb, adapt to or rapidly recover from disruptive
events (Francis, 2014).
8
1.4 Dissertation outline
The rest of the dissertation is organized as follows:
Chapter 2 presents background information, specifically the proposed definitions of terms
necessary to present the research of this thesis: Namely, it provides the definition of a critical
infrastructure, infrastructure dependency, the notion of failure in critical infrastructures along
with its definitions of impact and likelihood. In addition, definitions are given about the lower
granularity research in software systems. The following terms are defined: An Application
under Test, software flaw, logical error, business logic, static analysis, dynamic analysis and
invariants.
This chapter also includes a survey of the most common tools and frameworks currently
available both for software application code analysis and for modeling complex infrastructure
interdependencies. The chapter also presents the current state of academic work in critical
infrastructure dependency analysis and software application code analysis.
Chapter 3 presents the proposed methodology for detecting source code logical errors in
software that may lead to exploitable vulnerabilities that affect the stability of information
systems. It presents a complete methodology based on (Felmetsger, 2010), along with its
implementation in a tool named PLATO that identifies various types of logical errors in source
code and classifies them according to the way they influence an AUT and the position where
they manifest inside the source code. Preliminary results suggest that various types of logical
errors that lead to software vulnerabilities can be automatically detected using the presented
methodology. The chapter ends with a discussion on potential uses of the aforementioned
method and tool to areas beyond the protection of software that handles machinery in CIs. This
technique can provide possible alternatives to code auditing in the generic area of source code
analysis. Also, PLATO’s rankings of detections are discussed as potential findings that can be
combined with information from classic Risk Assessments to provide an impact estimation of
a specific subtype of failure in critical infrastructures; e.g. in CIs that use software to manipulate
machinery.
Chapter 4 includes a methodology developed for critical infrastructure dependency analysis
which uses dynamic time-based analysis models. The proposed methodology uses different
failure models and applies fuzzy logic to simulate realistic approximations of dynamic
cascading failures between infrastructures. We implemented the methodology in a tool called
CIDA which is able to simulate the effects of dynamic, large-scale and major disasters.
Preliminary results suggests that, given adequate input from Risk Assessments, CIDA is
capable of foretelling the most critical risk paths of interconnected infrastructures in various
Introduction
9
time points between the manifestation of a failure and critical time point; i.e. when it manifests
its highest impact on an infrastructure.
In Chapter 5, this thesis explores how graph centrality metrics can be applied to design and
to evaluate the effectiveness of risk mitigation strategies in dependency risk graphs made by
interdepended CIs, by pinpointing Critical Infrastructures of high dependency value amongst
interconnected infrastructures. The presented algorithm is able to explore the relation between
dependency risk paths using graph centrality characteristics and proposed critical
infrastructures in networks that are critical in the overall stability of the infrastructure web.
Preliminary results suggest that the proposed algorithm performs better than some well-known
techniques in pinpointing dangerous infrastructure links in interdepended CIs.
Chapter 6 concludes the dissertation with a summary, concluding remarks and future work.
10
(this page is intentionally left blank)
Background
11
Chapter 2: Background This chapter includes the definition of the term Critical Infrastructure, Risk and Risk analysis
that is adopted in this thesis for all granularity levels. It includes a risk management process for
every type of risk analysis process, as applied to various aspects of a critical infrastructure; both
in its information systems and its dependencies and overall threats. It continues with a
discussion of the security issues and currently proposed solutions for analyzing risk in CI’s
operating systems and presents some related literature on information systems and
infrastructure security. The chapter concludes with this thesis’s adopted threat model.
2.1 Definitions of Critical Infrastructure
This section reviews definitions used by governments in the context of national or regional
infrastructure protection programs and results are listed below (Gordon, 2008) (Emergency
management Australia, 2003) (Ministry of the Interior and Kindgom relations, 2008):
Table 1: National Definitions of Critical Infrastructure
Australia “Critical infrastructure is defined as those physical facilities, supply
chains, information technologies and communication networks which, if
destroyed, degraded or rendered unavailable for an extended period, would
significantly impact on the social or economic well-being of the nation, or
affect Australia’s ability to conduct national defence and ensure national
security.”
Canada “Canada’s critical infrastructure consists of those physical and
information technology facilities, networks, services and assets which, if
disrupted or destroyed, would have a serious impact on the health, safety,
security or economic well-being of Canadians or the effective functioning
of governments in Canada.”
Germany “Critical infrastructures are organisations and facilities of major
importance to the community whose failure or impairment would cause a
sustained shortage of supplies, significant disruptions to public order or
other dramatic consequences.”
Netherlands “Critical infrastructure refers to products, services and the accompanying
processes that, in the event of disruption or failure, could cause major social
12
disturbance. This could be in the form of tremendous casualties and severe
economic damage… ”
United
Kingdom
“The [Critical National Infrastructure] comprises those assets, services
and systems that support the economic, political and social life of the UK
whose importance is such that loss could: 1) cause large-scale loss of life;
2) have a serious impact on the national economy; 3) have other grave social
consequences for the community; or 3) be of immediate concern to the
national government.”
United
States
The general definition of critical infrastructure in the overall US critical
infrastructure plan is: "systems and assets, whether physical or virtual, so
vital to the United States that the incapacity or destruction of such systems
and assets would have a debilitating impact on security, national economic
security, national public health or safety, or any combination of those
matters." For investment policy purposes, this definition is narrower:
“systems and assets, whether physical or virtual, so vital to the United States
that the incapacity or destruction of such systems and assets would have a
debilitating impact on national security."
Table 1 above shows the definitions of critical infrastructure used in six (6) published critical
infrastructure protection plans or strategies. This review of definitions considers separately the
two words -“critical” and “infrastructure”- and then looks at the sectorial coverage of critical
infrastructure protection programs.
• Critical: In most countries‟ definitions, the word “critical” refers to infrastructure that
provides an essential support for economic and social well-being, for public safety and for the
functioning of key government responsibilities. For example, Canada’s definition of criticality
involves “serious impact on the health, safety, security, or economic well-being of Canadians
or the effective functioning of governments in Canada.” Germany refers to “significant
disruptions to public order or other dramatic consequences.” The Netherlands‟ critical
infrastructure policy refers to infrastructure whose disruption would cause “major social
disturbance”, “tremendous loss of life” and “economic damage”. Thus, the word “critical”
refers to infrastructure which, if disabled or destroyed, would result in catastrophic and far-
reaching damage.
• Infrastructure: The definitions of “infrastructure” used in official descriptions of critical
infrastructure tend to be broad. All 6 governments in Table 1 refer to physical infrastructure.
Background
13
Most also include intangible assets and/or to production or communications networks.
Australia, for example, refers to “physical facilities, supply chains, information technologies,
and communications networks.” Canada refers to “physical and information technology
facilities, networks, services and assets”. The United Kingdom refers to “assets, services and
systems”.
2.2 Risk Analysis in all granularity levels
Everyone takes and manages risks all the time, balancing potential rewards against uncertain
losses. Risk management remains nevertheless a very difficult process. It requires combining
the ‘‘hard’’ scientist’s approach, who treats risks as something that can be objectively
measured, with the view of the ‘‘social’’ scientist who argues that risk is a fuzzy concept and
the propensity to take risks is in part culturally constructed (CRAMM User guide, 2005).
Risk is the effect of uncertainty on objects (ISO/IEC, 2014). It is the chance of something
going wrong as a result of a hazard or a threat which has an impact on operations. Risks arise
out of uncertainty. A risk is measured in terms of its likelihood of happening and the
consequences if it should happen.
Risk management is the coordinated activity to direct and control an organization with regard
to risk (ISO/IEC, 2014). According to (CRAMM User guide, 2005) it is balancing the cost of
avoiding, reducing, transferring or accepting a risk with the consequences that can be expected
from taking the risk. The term risk management is used in a wide variety of disciplines, and
itself also combines concepts and techniques from a range of fields like statistics, economics,
operations research and decision theory.
Unfortunately, there are no standards for defining vulnerabilities and threat-sources,
assigning and combining impact and probability ratings, or introducing the impact of controls
in the field of information security related risk management. Recent standards and
recommendations on the management of information systems and organizing the protection of
information security within an organization widely recognize the importance of information
security related risk management.
According to (CRAMM User guide, 2005) Risk management processes typically include the
following four steps:
A. Establish the Scope
The first step in any risk management process consists in defining the scope of the risk
management process, in other words the information system that is the target of the evaluation,
its boundaries and environment. The characterization of the system must be as complete as
possible and most often includes the following elements:
14
• Hardware (e.g. servers, workstations, network equipment);
• Software (e.g. operating systems, system services, application software);
• Connectivity (internal and external);
• The information system’s mission;
• The information that is managed by the system and its requirements regarding
availability, integrity and confidentiality;
• Support staff and users; and
• Existing controls: technical controls (e.g. user identification and authentication
equipment, encryption hardware and software), management controls (e.g. security policy,
acceptable use policy), operational controls (e.g. backup and contingency operations, off-site
storage, user account creation and deletion procedures), physical security environment (e.g. site
security, data center policies), environmental security (e.g. controls for power, temperature,
humidity).
B. Identify the Risks
The second step of the risk management process consists in establishing a list of the risks to
which the information system is exposed.
First, based on the system and context description available at the end of the previous step,
the vulnerabilities that apply to the target of the evaluation are identified. A vulnerability is a
weakness of an asset or control that can be exploited by one or more threats (ISO/IEC, 2014).
It can be expressed as any flaw or weakness in the design of a system, in its implementation or
in the controls that are in place to protect it and that can result in damage when it is accidentally
triggered or intentionally exploited.
A threat-source is either the combination of the intent and the means to intentionally exploit
a vulnerability (e.g. a thief, a disgruntled employee) or a situation that may accidentally trigger
a vulnerability (e.g. an earthquake, a sloppy user) (CRAMM User guide, 2005). A threat is a
potential cause of an unwanted incident, which may result in harm to a system or organization
(ISO/IEC, 2014). It is the potential for a threat-source to accidentally trigger or intentionally
exploit vulnerability. When for a given vulnerability there is no threat-source that has the
technical ability or motivation to exploit it, there is no threat (CRAMM User guide, 2005).
Likewise, when there is no vulnerability present for which a given threat-source has the
necessary skills, time and budget, this threat source poses no threat.
Each threat is after that matched with the list of controls that were identified in the first phase,
and that mitigate the likelihood of a vulnerability being exercised or reduce the impact of such
Background
15
an adverse event when it occurs. The resulting tuple (threat, threat-source, list of relevant
controls) defines the risk that will be assessed and treated in the subsequent steps.
C. Analyze the Risks
In this step, the risks that were identified, are to be analyzed in more detail, so that the minor,
acceptable risks can be separated from the major risks which must absolutely be eliminated or
reduced. This involves deriving for each risk an overall likelihood rating that indicates the
probability that the vulnerability may be exercised by the corresponding threat-source. The
second element in risk assessment is trying to rate the adverse impact of the vulnerability when
it were to be exercised. This rating will be based on an evaluation of the loss or degradation of
integrity, availability, and confidentiality of the information that is threatened by the
vulnerability.
When determining the probability and impact of a threat, the existing controls that reduce
the likelihood or impact and their adequacy have to be taken into account. The combination of
probability and impact will finally be translated into a single level of risk to the information
system, for instance using a risk-level matrix.
D. Treat the Risks
Risks can be handled in a number of ways (CRAMM User guide, 2005):
• Risk Avoidance: means simply not performing the activity that carries the risk.
Unfortunately this also typically means losing out on the potential gain that performing the
activity might have produced.
• Risk Reduction: involves approaches that reduce the probability of the vulnerability
being triggered or reduce the impact when the vulnerability is triggered. Reducing a risk most
often involves putting in place controls.
• Risk Transfer: means passing the risk on to another party that is willing to accept the
risk, typically by contract or by hedging. Insurance is an example of risk transfer using
contracts.
• Risk Retention: means accepting the loss when it occurs. Risk retention is a viable
strategy for small-impact risks where the cost of insuring against the risk would be greater over
time than the total losses sustained. Also, all risks that are not avoided nor transferred, and that
one does not can or wish to reduce any further, automatically fall under this category. This
includes risks that are so large or catastrophic that they either cannot be insured against or the
premiums would be infeasible.
The combination of methods used to handle each of the risks that were identified, analyzed
and treated, leads to a risk management plan that must then be implemented. Risk management
can be performed once for a given system, for instance before it comes in operation, and then
16
periodically updated during the lifetime of the system. The back coupling is in this case not
permanent but rather periodically triggered. Risks management can however also be conceived
as a continuous process and influence decision-making at all instances through the life of the
system.
2.4 Threat model
This section briefly introduces the threat model for critical infrastructures, which includes
usage and threat details. Unless stated differently, the following chapters and the presented
methods always refer to or are based on an input point of the following model. This means that
the scope of this thesis focuses specifically on two points of entry for the model, as depicted
above while analyzing the scope of risk analysis: Software and (external) connectivity. The
methods presented in the following chapters, thus, try to cope with inherent and unresolved
security issues on these two elements.
Users. In this work users are considered to be average users, i.e. not security nor technical
experts. Regarding their practices it is assumed that these users:
a) Use software to manipulate various types of functionality (e.g. infrastructure machinery,
ticketing systems etc.),
b) Utilize software only within the boundaries of their given rights,
c) Do not intentionally try to sabotage, misuse or break software or hardware under their
supervision,
The later, is in accordance with the expectations for a user without a security mindset and
without advanced technical skills.
Threats. This work focuses on two types of threats, following the corresponding elements
of risk analysis; (i) threats relevant to the (mis)use of software applications by average users
and (ii) threats manifesting due to external interdependencies between critical infrastructures.
This work also focuses on the cascading failures through software malfunction threat, since as
discussed earlier, the use of software to control machinery in critical infrastructures can create
the conditions necessary for cascading failures to manifest. The decision to focus on these
threats is validated both by subsequent industrial incidents and relevant academic publications,
depicting the dangers of software misuse in IT systems and also the danger of interconnecting
multiple infrastructures.
The modeling and analysis of interdependencies between critical infrastructure elements is a
relatively new and very important field of study (Pederson, 2006). Much effort is currently
Background
17
being spent to develop models that accurately simulate critical infrastructure behavior and
identify interdependencies and vulnerabilities. The results of these simulations are used by
private companies, government agencies, military, and communities to plan for expansion,
reduce costs, enhance redundancy, improve traffic flow, and to prepare for and respond to
emergencies.
Modern infrastructures are often depended on other infrastructures to function properly. This
necessity has led to the development of complex networks of interdepended infrastructures.
These dependency graphs of hide information about what will happen if a failure occurs; in
other words, they are as safe as their most critical path of interdependencies and as exposed as
their most dangerous node. For example, as far as the dangers of interdepended infrastructures
is concerned, Rinaldi, Peerenboom and Kelly in (Rinaldi et al., 2001) provide a visual
presentation of the well-known electric failure scenario of California which is a characteristic,
real-case example of a multi-order dependency between CIs. The electric power disruptions in
California caused cross-sectoral cascading effects, as power disruptions affected natural gas
production, operation of petroleum product pipelines transporting gasoline and jet fuel along
with the operation of massive water pumps for crop irrigation.
Concerning software and its dangers in Critical Infrastructure information systems, one
should look no further than the incident with the security worm, Stuxnet. The Stuxnet incident
was a typical example of software being able to misuse functionality in machinery and manifest
catastrophic failures across multiple infrastructures. Many Critical Infrastructures use Remote
Terminal Units (RTUs) and Programmable Logic Controllers (PLCs) as control locations, in
order to handle the machinery and functionality of an infrastructure (e.g. valves, sensors,
breakers, etc.). Thus, a failure on any one of them may affect the operation of the entire
infrastructure and start a cascading event, where multiple CIs fail due to their dependencies.
When industries connected SCADA systems with the Internet, these control systems were
exposed to various vulnerabilities and threats (Krutz, 2005). Built as stand‐ alone, isolated
control systems, they lacked the proper security measures needed to support a robust and safe
functionality over the Internet. For example, an over-the-Internet, man-in-the-middle attack on
a green diesel generator that led to a total meltdown has been presented in the literature
(Alcaraz, 2011).
2.5 Summary
This chapter presented a definition of the terms Risk, Risk analysis, Critical Infrastructure
and infrastructure threats w.r.t. their type. It also included a risk analysis methodology, which
includes the basic concepts for identifying threats and input vectors for failure manifestation.
18
Furthermore, the installation of the majority of the proposed security controls requires advanced
technical skills from the users. Finally, the chapter ends with a discussion of the thesis’ threat
model.
Background
19
(this page is intentionally left blank)
20
Chapter 3: Automatically detecting logical errors,
race conditions and IT infrastructure failures in
software using dynamic invariant assertions and
symbolic execution
3.1 Introduction
The proliferation The sum of all functional requirements of an application reflect the intended
program behavior; that is, what the programmer wants his code to do and what not to do. During
software development, functional requirements are translated into source code. A software error
or fault is the difference between a computed, observed, or measured value and the true,
specified or theoretically correct value or condition inside the software code (peng, 1993). A
software vulnerability is the characteristics and degree of the fatal software-failure occurrence
caused by an error (Kimura, 2006).
Research on automated detection of software errors and vulnerabilities has mainly focused
on static analysis and software model checking techniques that are effective in detecting a priori
specified errors (e.g. Time Of Check - Time Of Use errors, null pointer dereferences etc.), bad
coding patterns and some types of exploitable vulnerabilities (e.g. unsanitized input data, buffer
overflows etc.) (Codepro 2015) (Ucdetector, 2015) (Pmd, 2015) (Hovemeyer, 2004) (Coverity,
2015) (Pasareanu, 2004) (Java pathfinder tool, 2015). Yet, errors related to the intended
program functionality, which are broadly called logical errors, are not a priori known. In a code
auditing process, they cannot be analyzed as pattern-specific errors since they are rather
application-specific. At the level of the program’s execution flow, these errors will cause
execution diversions that manifest as unintended program behaviour (Felmetsger, 2010). A
typical example is the following from (Felmetsger, 2010): ”a web store application allows,
using coupons, to obtain a one-time-discount-per-coupon on certain items; a faulty
implementation can lead to using the same coupon multiple times, thus eventually zeroing the
price (e.g. by pressing the ”BACK” button, then the ”FORWARD” one and re-entering a
coupon code in the form)”.
Since logical errors in an Application under Test (AUT) are essentially execution
deviations from its intended functionality, their automated detection needs to be based on
some model of the AUT’s operational logic. Such a model can be inferred in the form of
likely invariants from the dynamic analysis of official executions of the AUT’s functionality
(i.e. execution of scenarios). Dynamic invariants are properties that are likely true at a certain
point or points of the program and, in effect, reveal information about the goal behaviour,
Mitigation of web threats
21
the particular implementation and the environment (inputs) under which the program runs
(Ernst, 2007). The method to detect logical errors proposed in this article evaluates inferred
dynamic invariants in the form of assertions using multiple symbolic executions of an AUT,
in an effort to detect various types of logical errors. Analysis of assertions of automatically
inferred dynamic invariants cannot be neither sound, nor complete, and a proper post-
processing is necessary, in order to derive a usable result. We note that we do not propose
such an approach as an alternative to the mainstream static analysis techniques, which in any
case target a different type of errors and they cannot detect logical errors.
Our method for the automated detection of logical errors has been implemented in a
prototype tool called PLATO and consists of the following steps:
1. For an AUT, a representation of its operational behavior is generated in the form of
dynamic invariants. These invariants are collected by dynamic analysis of the AUT using
the Daikon tool (Ernst, 2007).
2. Through a source code analysis of the AUT and its Abstract Syntax Tree we gather: (i)
instructions that play a key role in multiple software vulnerabilities (as they are classified
in (CWE, 2015); and (ii) execution path branching points.
3. Dynamic invariants from step 1 are filtered using the information from step 2. PLATO
keeps only the invariants related to variables that affect the execution flow or are used in
“dangerous” instructions from those mentioned in step 2. These invariants are then
instrumented as assertions into the AUT’s source code.
PLATO then relies on custom-made extensions of the Java PathFinder (JPF) tool (Pasareanu
2004) (JPF, 2015) to verify the asserted invariants through the AUT’s symbolic execution. JPF
symbolically executes the AUT covering multiple execution paths. The traversed paths must
adequately cover all intended functionality to be tested: if an assertion is found that is satisfied
in an execution path, while it is simultaneously violated in another path with the same prefix,
then a possible logical error is flagged.
A wide range of errors that are classified as logical (as seen in (CWE, 2015)) are detected by
flagging suspicious invariants this way. For example, invariant violations on variables involved
in race conditions allow PLATO to detect race violations between threads.
PLATO ranks detections (i.e flagged assertions) using a classification approach that
combines the output from steps 3 and 4. Ranking helps differentiate the danger level of
detections and can aid in mitigating the problem of possible false positives. Each detection and
the corresponding execution path are ranked using two fuzzy membership classifiers, namely
Severity and Reliability. Severity ranking is based on a classification technique called
Information Gain. Reliability is based on a software analysis technique called Computational
Density. PLATO’s Fuzzy Logic system for ranking the overall Risk of detections combines
22
output from these two classifiers using linguistic variables and outputs an informative
classification rank.
The described analysis framework is based on a method that was first proposed in
(Felmetsger, 2010) and then extended in (Stergiopoulos, 2012) (Stergiopoulos, 2013)
(Stergiopoulos, 2014), but only basic concepts from those works are kept. In PLATO, dynamic
invariants are evaluated using different techniques and the tool is now capable of analyzing the
full range of instrumented invariants. The fuzzy logic classification technique from
(Stergiopoulos, 2012) (Stergiopoulos, 2013) (Stergiopoulos, 2014) is still used, but its present
version is comprised of two new formal classifiers, which are introduced in place of the
previously used empirical rules.
Finally, the chapter concludes with a discussion of the results in Section 3.7 and a summary
in Section 3.8.
3.2 Related Work
Recent developments in debugging techniques also focus on the detection of logical errors,
but they do not aim to a fully automated program analysis. Delta debugging (Zeller, 2002) is a
state-altering technique that systematically narrows the difference between the states of a failed
program run from the states of a failure-free run, down to a small set of variables. The intuition
is that any difference between the two execution paths could be the failure cause. Predicate
switching is a variant of delta debugging (Zhang, 2006) that alters predicate truth values during
of a program execution. Given a failing execution, the goal is to find the predicate that, if
switched from false to true or the opposite, it causes the program to execute successfully. A
limitation of state-altering techniques is that they do not address the problem of semantic
consistency; there is no guarantee that by altering a state the new execution path will still be a
valid program run (Baah). A second limitation is the usability of this technique, since the
program has to re-run after every single state alternation. In our approach for detecting logical
errors, state alternation is avoided through the use of dynamic invariants along with a one-time
symbolic execution of the AUT.
Another class of techniques to detect logical errors is based on program slicing. In (Weiser,
1981), the authors introduce such a technique for debugging programs. A drawback is that the
slice set often contains too many program entities and in some cases the whole program. In
(Zhang, 2006), a technique is proposed using a threshold to prune the backward slice computed
for a particular program entity. By pruning the backward slice, the size of the computed slice
set is reduced. Slicing techniques do not reflect the strength of dependencies between the
program entities and how likely is for each of them to be the cause of a failure. Moreover, they
Mitigation of web threats
23
only compute program entities that are associated with failures and they do not detect which
program entities cause those failures (Baah). Finally, these techniques do not provide any
information for how to start searching for the fault.
In (Doupe, 2011), the authors focus exclusively on the detection of specific flaws found in
web applications, whereas in (Balzarotti, 2007) web applications are analyzed for multi-module
vulnerabilities using a combination of analysis techniques. However, both works do not address
the problem of profiling the source code behavior or 145 detecting logical errors per se.
Certain types of logical errors in web applications can be detected with the approach
discussed in (Felmetsger, 2010). A set of likely invariants that characterize the execution of the
AUTs is inferred using the Daikon tool (Ernst, 2007) (Daikon tool, 2015). The Daikon results
are then used in JPF (Pasareanu, 2004) (Java pathfinder, 2015) to model check the behavior of
the AUT over symbolic input. However, it is not shown how the proposed approach scales to
larger, standalone applications.
Variants of our method were presented in (Stergiopoulos, 2012) and (Stergiopoulos, 2014).
In (Stergiopoulos, 2012), we specifically targeted logical errors in GUI applications. We
described a preliminary deployment of a Fuzzy Logic ranking system to mitigate the possibility
of false positives and we applied the method on lab test-beds. In (Stergiopoulos, 2013), the
Fuzzy Logic ranking system was formally defined and further developed. In this work, the
method that we first proposed in (Stergiopoulos, 2012) (Stergiopoulos, 2013) (Stergiopoulos,
2014), is evolved to a more complete and effective approach with the capacity to be applied to
real-world, complex applications, instead of test-beds and simple GUI AUTs.
3.3 Analysis Building Blocks
In this section, the main building blocks of PLATO’s methodology are described, namely:
(i) how the behaviour of an AUT is modeled using likely dynamic invariants, (ii) how the
obtained likely invariants are verified through symbolically executing the AUT and (iii) how
the results are classified using fuzzy logic to measure the impact and the size/complexity of the
affected code, for each detection.
3.3.1. Dynamic Invariants for profiling the behavior of the source code
The functionality of an AUT is captured in the form of dynamic invariants, generated by the
Daikon tool (Ernst, 2007) (Daikon invariant manual, 2015). These invariants are logical rules
for variables (e.g. p!=null or var=="string") that hold true at certain point(s) of a
program in all monitored executions.
The generated dynamic invariants can reflect the intended functionality of the AUT, if they
are derived from monitored executions of representative use-case scenarios. Existing
24
functionality restrictions and prerequisites also have to be taken into account while using
Daikon.
The validity of the inferred dynamic invariants (i.e. the inferred program behaviour) is tested
against as many execution paths of the AUT as possible, using symbolic execution of the AUT.
Intuitively, if there is an execution path, which violates a (combination of) dynamic invariant(s),
then a logical error may exist, which affects the variable(s) referred in the invariant.
3.3.2. Symbolic execution for verifying dynamic invariants
PLATO converts the likely invariants into Java assertions and instruments them into the
source code. For example, let us consider that the invariant ”p != null” holds true when
the execution flow enters a method. In this case, PLATO creates the assertion [assert (p
!= null);] and instruments it at the beginning of that method, just before any other
instruction.
Likely dynamic invariants are instrumented according to two filtering criteria: invariants
concerning variables which affect the execution flow and invariants related to instructions
which are tied to known application vulnerabilities (Marin, 2008) (Harold, 2006). For the
former, we particularly focus on the conditional expressions in branches and loops. The latter
is implemented by using a taxonomy that classifies source code instructions according to their
danger level. This taxonomy is embedded in PLATO and is based on the taxonomies presented
in (Martin, 2006), the Oracle’s Java Taxonomy (Gosling, 2014) (Java Platform, 2015) and
reports from code audits (Hovemeyer, 2004). More information on this taxonomy is provided
in Section 4.3.1, which covers technical details.
Daikon’s invariants are then cross-checked with a set of finite execution paths and their
variable valuations for each tested path. For this purpose, PLATO obviously needs execution
paths that adequately cover the functionality of the AUT.
We introduce here formal definitions for the profiling of a program’s behavior and we discuss
how they are related to the implementation of PLATO. An imperative program P = (X, L, ℓ0,
T) defines (Jhala, 2009) a set X of typed variables, a set L of control locations, an initial location
ℓ0 ∈ L, and a set T of transitions. Each transition τ ∈ T is a tuple (ℓ,ρ,ℓ′), where ℓ,ℓ′ ∈ L are control
locations, and ρ is a constraint over free variables defined in X ∪X′, with X representing the
program’s state variables at control location ℓ and X′ the state variables at control location ℓ’.
The set L of control locations is comprised of the source code points which control the
execution flow or are critical to program functionality, i.e. all conditional expressions in
branches and loops along with instructions that manipulate user input data. This is in line with
NIST’s recommendation [31], which states that the impact of a source code point in a program’s
Mitigation of web threats
25
execution may be captured by: (i) the program’s entry points and variables with user data, called
Input Vectors (ii) the Branch Conditions (e.g. conditionals used, for example, in if-statements)
and (iii) program points where instructions use variables with user data (Sinks).
The state of P is a valuation of the variables in X at some control location during the program
execution. The set of all possible states for P is denoted by u.X. We shall represent sets of states
using constraints. For a constraint ρ over X ∪ X′ and a pair of states (s, s′) ∈ u.X × u.X′, we write
(s, s′) |= ρ if the variables valuations in X and X′ satisfy the constraint ρ. We focus on AUTs with
an explicitly defined initial state that assigns values to all variables in X. A finite computation
of the program P is any sequence (ℓ0, s0),(ℓ1, s1),...,(ℓk, sk) ∈ (L×u.X), where ℓ0 is the initial
location, s0 is the initial state, and for each i ∈ {0,...,k1}, there is a transition (ℓi,ρ, ℓi+1) ∈ T such
that (si, si+1) |= ρ. A location ℓ is reachable if there exists a state s, such that (ℓ, s) appears in
some computation. An execution path or, simply, path of the program P is any sequence π =
(ℓ0, ρ0, ℓ1), (ℓ1, ρ1, ℓ2),...,(ℓk−1, ρk−1, ℓk) of transitions.
PLATO monitors program states and invariant assertions in execution paths by implementing
a state listener called PlatoListener inside the JPF tool. Our listener keeps a log of all the
program’s constraints ρi (assertions) along with their valuations for the first execution path and
for the rest of them, while the program’s execution branches due to JPF’s backtracking to
previous states, in order to execute different transitions.
Definition 1. A logical error manifests if there are execution paths πi and πj with the same
prefix, such that for some k ≥ 0 the transition (ℓk, ρk, ℓk+1) results in states (ℓk+1, si),(ℓk+1, sj) with
si = sj and for the dynamic invariant rk, (si−1, si) |= rk in πi and (sj−1, sj) |= rk in πj, i.e. rk is satisfied
in πi and is violated in πj.
So, for each dynamic invariant rk, PlatoListener relies on JPF’s symbolic execution to
gather all execution paths that evaluate the dynamic invariant. Then, for each path with some
state sj such that (sj−1 ,sj) |= rk, PLATO compares it with other paths with the same prefix such
that si = sj and (si−1, si) |= rk.
3.3.3. Fuzzy Logic classification of detections
The criterion introduced in Definition 1 can only indicate a logical error and it cannot be
used by itself to safely characterize any detection. First, the dynamic analysis is neither sound
nor complete and it is rather unwise to assume that upon detection, a logical error is located at
the program point, where an invariant was violated. The exact localization of the error
inevitably depends on the intervention of human ingenuity, as opposed to its manifestation,
which is automatically detected. Moreover, it is not true that all the logical errors can divert the
programs’ execution to exploitable states and that they have comparable impact on the
26
functionality of an AUT. Motivated by these facts, we recognize the need to assess invariant
assertion violations based on various characteristics and their position within the source code.
Similarly to a code auditor’s reasoning upon detection of a flaw in the AUT’s functionality,
PLATO classifies the found detections using a fuzzy set theory approach combined with two
advanced classification functions. For each detection, the result is a quantitative truth value
ranging from 0 to 1, which measures the error’s impact on the AUT’s execution. A quantitative
truth value corresponds to a membership percentage in a fuzzy set (Perfilieva, 1999) with the
selected fuzzy sets describing different levels of impact (danger). As an example, a logical error
that doesn’t affect critical functionality and does not divert the AUT’s execution will not be
characterized as a high impact detection, whilst a logical error in an if-statement of a user
authentication module that can lead to exploitable behaviour should, therefore, be classified as
extremely dangerous. Thus, every assertion violation along with the corresponding execution
path are classified into two different groups of sets:
the Severity sets, which quantify the danger level of the execution path, i.e. the impact
that an exploitable error would have, if it would be manifested on that path;
the Reliability sets, which quantify the overall reliability of an execution path based on
the size and the complexity of the code traversed in it (a code metric is used named
Cyclomatic Density).
With this fuzzy logic approach, we also aim to confront two inherent problems in automated
logical error detection: the large data sets of the processed AUT execution paths and the
possible false positives. Regarding the first mentioned problem, PLATO helps the code auditor
to focus only to those path transitions that appear having high ratings in the classification
system. Regarding the false positives, we note that error detections cannot be validated against
predefined error patterns. PLATO’s ratings are therefore used to implement criteria that take
into account the possibility of a true logical error in some transition. The following subsections
will help clarify how this takes place.
3.3.3.1. Severity
For an execution path π, Severity(π) measures π’s membership degree in a Severity fuzzy set
that reflects how dangerous is a flaw if it were to manifest in path π, i.e. its relative impact.
Execution path π is weighted based on how its transitions and corresponding code instructions
affect the program’s execution: if there are transitions in the path that are known to manifest
exploitable behaviour, then π is considered dangerous and is assigned higher Severity ranks.
Definition 2. Given the execution path π, we define
Mitigation of web threats
27
Severity(π) = ν ∈ [1,5]
to measure the severity of π on a Likert-type scale from 1 to 5.
Likert scales are a convenient way to quantify facts (Albaum, 1997) that, in our case, refer
to a program’s flow. If an exploitable behaviour were to manifest in an execution path, the
scale-range captures the intensity of its impact in the program’s control flow. In order to weight
paths, Severity is based on the Statistical Information Gain, a measure used to classify execution
paths in one out of five Severity categories that are ranked from one to five. Categories are then
grouped into Fuzzy Logic sets using labels: high severity (4-5), medium (3) or low (1 or 2).
3.3.3.2. Measuring Severity of execution paths using its Statistical Information Gain
When a code auditor inspects a program, he traces execution paths with specific
characteristics trying to find flaws in them. In most cases, the traced paths include: (i) the
program’s input vectors, (ii) branch conditions with variable checks and (iii) sinks, i.e.
instructions that should only use sanitized data.
Our Severity classification approach is based on the Expected Information Gain (aka
Expected Entropy Loss) statistical measure (Abramson, 1964) that has been successful in
feature selection for information retrieval (Etzkorn, 1997). Feature selection is used to improve
the effectiveness and efficiency during classification, since it removes non-informative terms
according to corpus statistics (Yang, 1997). The Information Gain has been used before by
Glover et al. (Glover, 2001) and Ugurel et al. (Ugurel, 2002) for classifying source code. Here,
we use it to classify execution paths and their corresponding source code instructions into
danger levels.
To measure the Expected Information Gain of an execution path, we need characteristics
(features) to look for. PLATO uses a taxonomy of dangerous source code instructions. These
instructions are tied to known vulnerability types (Martin, 2008) (National Vulnerability
database, 2015). The taxonomy is divided into 5 subsets of instructions that act as Feature Sets
to classify execution paths. Each subset’s source code instructions are considered to have the
same impact level (i.e. they are known to be involved in similar types of vulnerabilities). Each
set is characterized by a number on the Likert scale (1 to 5) depicting the danger level of its
code instructions: Set 1 contains the least dangerous instructions while Set 5 contains the most
dangerous source code instructions, known to be involved in many critical vulnerabilities. For
example, the exec() instruction is known to be tied to OS injection vulnerabilities (Martin,
2008). Therefore exec() is grouped in Set 5 of the taxonomy.
Severity ratings are applied by classifying each execution path into one of these five Severity
feature sets, which correspond to specific impact levels. For each execution path, every Feature
28
Set gets its own Information Gain measurement. High information gain values for a specific
feature set means that this set is an effective discriminator of an execution path (Ugurel, 2002).
Thus, feature set measurements act as execution path classifiers: they rank paths into different
danger levels. The feature set with the highest Expected Information Gain classifies the
execution path in the danger level category that it represents.
In the following paragraphs, we provide a brief description of this theory (Abramson, 1964).
Let Pr(C) be the probability of a transition in the path that indicates that the path is considered
dangerous. Pr(C) is quantified as the ratio of the dangerous instructions over the total number
of instructions in the path. Let f be the event that a specific instruction exists in the path. We
also denote by C and 𝑓 the negations of C and f.
The prior entropy e is the probability distribution that expresses how certain we are that an
execution path is considered dangerous, before feature f is taken into account:
𝑒 = −𝑃𝑟(𝐶)𝑙𝑔𝑃𝑟(𝐶) − 𝑃𝑟(C)𝑙𝑔𝑃𝑟(C) (1)
where lg is the binary logarithm (logarithm to the base 2). The posterior entropy, when
feature f has been detected in the path is
𝑒𝑓 = −𝑃𝑟(𝐶|𝑓)𝑙𝑔𝑃𝑟(𝐶|𝑓) − 𝑃𝑟(C|𝑓)𝑙𝑔𝑃𝑟(C|𝑓) (2)
whereas the posterior entropy, when the feature is absent is
𝑒�� = −𝑃𝑟(𝐶|𝑓)𝑙𝑔𝑃𝑟(𝐶|𝑓) − 𝑃𝑟(C|𝑓)𝑙𝑔𝑃𝑟(C|𝑓) (3)
Thus, the expected overall posterior entropy (EOPE) is given by
𝐸𝑂𝑃𝐸 = 𝑒𝑓𝑃𝑟(𝑓) + 𝑒��𝑃𝑟(𝑓) (4)
and the expected Information Gain (EIG) for a given feature f is
𝐸𝐼𝐺 = 𝑒 − 𝑒𝑓𝑃𝑟(𝑓) − 𝑒��𝑃𝑟(𝑓) (5)
The Expected Information Gain is always non-negative and higher scores indicate more
discriminatory features. The higher the EIG for a given feature set of instructions f, the more
certain we are that this set f best describes the execution path.
Similarly to (Ugurel, 2002), EIG is calculated based on ratios between path instructions
considered dangerous (e.g. execution instructions like exec()) and the total number of
instructions in the transitions of each execution path. A taxonomy of Java instructions is the
base of instructions acting as feature sets (corresponding to the event f in the above equations).
Example instructions of the taxonomy and their classification into Feature Sets is given in Table
Mitigation of web threats
29
2 below, where some dangerous instructions are shown together with the feature set they belong
to. More technical details on the taxonomy are given in Section 4.3.1, whereas for a complete
list of all instructions, the reader can access the link at the end of the article.
Severity (π) basically tells us which feature set best characterizes a path π; the one that
exhibits the highest overall EIG. Since each feature set f is tied to a specific impact (danger)
level, then this level also indicates the danger level of the corresponding execution path.
Rank Example of classified methods Feature Set (Category level)
Low javax.servlet.http.Cookie Set 1 (Level 1)
Low java.lang.reflection.Field Set 2 (Level 2)
Medium java.io.PipedInputStream Set 3 (Level 3)
High java.io.FileInputStream Set 4 (Level 4)
High java.sql.ResultSet::getString Set 5 (Level 5)
Table 2: Severity classification examples
3.3.3.3. Reliability
As a measuring function, Reliability is used to classify execution paths into Reliability sets.
It quantifies how reliable an execution path is by computing the likelihood that an exploitable
behavior is manifested in a variable usage.
Definition 3. Given the execution path π, with a set of state variables, we define Reliability as
Reliability(π) = ν ∈ [1,5]
to measure the reliability of π on a Likert scale from 1 to 5.
Similarly to the Severity function, our fuzzy logic system classifies execution paths in
categories: high severity (4-5), medium (3) or low (1 or 2).
3.3.3.4. Measuring code Reliability with Cyclomatic Density
The inherent risk or risk build-up of an AUT is connected to its source code’s complexity
(Chhabra, 2014). A broadly accepted measure is the well-known Cyclomatic Complexity (Bray,
1997) that measures the maximum number of linearly independent circuits in a program’s
control flow graph (Gill, 1991). The original McCabe metric is defined as
V (G) = e − n + 2
30
where V(G) is the cyclomatic complexity of the flow graph G of a program, e is the number of
edges and n is the number of nodes in the graph. McCabe showed that V(G) can be computed
by applying the following steps [43]:
1. increment by one for every IF, CASE or other alternate execution construct;
2. increment by one for every DO, DO-WHILE or other repetitive construct;
3. add two less than the number of logical alternatives in a CASE;
4. add one for each logical operator (AND, OR) in an IF.
However, Cyclomatic Complexity does not take into consideration the size of the analyzed
code. Research conducted in the Software Assurance Technology Center of NASA has showed
that the most effective evaluation of the inherent risk of an AUT should be based on a
combination of the (cyclomatic) complexity and the code’s size (Rosenberg). Modules with
both a high complexity and a large size tend to have the lowest reliability. Modules with smaller
size and high complexity are also a reliability risk, because they feature very terse code, which
is difficult to change or to be modified.
To this end, PLATO implements heuristics that assign Reliability ratings to execution paths
through a cyclomatic density analysis. The proposed method is based on McCabe’s algorithm
and the computation of the Cyclomatic Density for each execution path. The Cyclomatic
Density is the ratio of the Cyclomatic Complexity to the logical lines-of-code, which measures
the number of executable “statements” in the path (some statements are excluded like for
example a variable assignment) (Using code quality metrics, 2015). This ratio represents the
normalized complexity of the source code of an execution path π and it is considered a
statistically significant single-value predictor of code’s maintainability (Using code quality
metrics, 2015). The higher the Cyclomatic density value, the denser the logic. Related research
(Using code quality metrics, 2015) (Rosenberg) proposes that Cyclomatic Density values for
the code to be simple and comprehensible should be in the range of .14 to .42.
Each path is assigned a density value. The higher the value, the more complex the logic of
the traversed code is and therefore more likely to have logical errors lurking in its instructions
[44, 45]. Table 3 depicts the classification categories for execution paths that can be applied
using the Reliability classification function.
Rank Example of classified methods Category
Low Cycl. Complexity Density <= 0.1 1
Mitigation of web threats
31
Low Cycl. Complexity Density >0.1 &&
Cycl. Complexity Density <= 0.2 2
Medium Cycl. Complexity Density >0.2 &&
Cycl. Complexity Density <= 0.3 3
High Cycl. Complexity Density >0.3 &&
Cycl. Complexity Density <= 0.4 4
High Cycl. Complexity Density >0.4 5
Table 3: Reliability categories based on Cyclomatic Density values
3.3.3.5. Risk: Combining Severity and Reliability ratings
According to OWASP, the standard risk formulation is an operation over the likelihood and
the impact of a finding (Martin, 2008):
Risk = Likelihood ∗ Impact
We adopt this notion of risk to enhance the logical error classification approach. For each
execution path π, an estimate of the associated risk is computed by combining Severity(π) and
Reliability(π). We opt for an aggregation function that allows taking into account membership
degrees in a Fuzzy Logic system (Foundation of fuzzy logic, 2015):
Definition 4. Given an AUT and an execution path π of a program P function, Risk(π) is the
aggregation
Risk(π) = aggreg(Severity(π),Reliability(π))
with a fuzzy set valuation
Risk(π) = {Severity(π)} ∩ {Reliability(π)}
Aggregation operations combine several fuzzy sets to produce a single fuzzy set. Risk ratings
have the following interpretation: for two execution paths π1 and π2, if Risk(π1) > Risk(π2), then
π1 is more dangerous than π2, in terms of the likelihood of these paths to divert execution to
non-intended states and to cause an unintended program behaviour. The risk of each execution
path is plotted separately, thus producing a numerical and a fuzzy result. By aggregating the
two membership sets (Severity and Reliability), a new membership set is produced. The
aggregated Risk membership value of an execution path in Risk sets is calculated using Fuzzy
Logic’s IF-THEN rules. An example is given in Figure 1.
32
Figure 1: Example of a Fuzzy Logic rule
PLATO then applies defuzzification (Leekwijck, 1999) on the aggregated Risk output using
the Center of Gravity technique: a single value is computed from the two fuzzy sets and their
corresponding membership degrees, i.e. the involvedness of each fuzzy set presented in Likert
values. In order to aid the end user, Severity and Reliability values are grouped into the same 3
sets as those in the Severity and Vulnerability scales (Low, Medium, High), with an
approximate width of each group of (5/3) = around 1,5 (final ranges: Low in [0, 2], Medium in
(2, 3.5] and High in (3.5, 5]). Figure 1 provides an example of how Risk is calculated using
Fuzzy Logic linguistic rules. Table 4 shows the fuzzy logic output for Risk, based on the
aggregation of Severity and Reliability.
Risk, Severity and Reliability ratings are supplementary to invariant violations and do not
provide the basic mechanism for logical error detection; they just provide a more clear view for
the code auditor. Also, high Severity rankings have more weight than Reliability rankings:
rightmost maximum values are taken into account during aggregation of sets that assign more
weight on Severity values. Rightmost maximum is found to have closer-to-the-truth ranking
results since Severity ratings take into consideration instructions in path transitions whilst
Reliability ratings provide only a generic view of the execution path’s overall complexity.
Severity /
Reliability Low Medium High
Low Low Low Medium
Medium Low Medium High
High Medium High High
Table 4: Severity x Reliability = R - Risk sets
The Fuzzy Logic system has been implemented using the jFuzzyLogic library (Cingolani,
2012). The technical details for the implementation of a fuzzy logic system in Java can be found
in (Cingolani, 2012), (Stergiopoulos, 2012) and (Stergiopoulos, 2013).
3.4 A method to detect logical errors in source code
Mitigation of web threats
33
3.4.1. The method’s workflow
The analysis building blocks described in section 3 and implemented in PLATO are part of
our workflow for logical error detection with the following steps:
1. Use case scenarios. We assume the existence of a test suite with use case scenarios that
exercises the functionality of the AUT. The selected use-case scenarios must cover the
intended AUT’s functionality to a sufficient degree. This can be quantified by appropriate
coverage metrics.
2. For each use-case scenario, a dynamic analysis with the Daikon tool is performed. A set
of inferred dynamic invariants is obtained that characterize the functionality of the AUT
based on the executed use case scenarios. 3. Daikon invariants are loaded in PLATO and
are processed as follows:
The inferred dynamic invariants are filtered by PLATO, in order to use only
those referring to high-risk transitions, i.e. (i) instructions that affect the
program’s execution flow, and (ii) instructions that are connected to the
manifestation of an exploitable behavior (e.g. for executing OS commands
with user input).
PLATO instruments the AUT code with the critical dynamic invariants,
which are embedded into the code as Java assertions (Martin, 2008).
3. The instrumented source code is symbolically executed in NASA’s JPF tool with our
PlatoListener extension. A sufficiently large number of feasible execution paths has
to be covered, far more than the initial use case scenarios covering the intended
functionality. JPF relies on the PlatoListener in order to check for existing assertion
violations and then flags the invariants satisfying the criterion of Definition 1.
4. PLATO gathers PlatoListener detections and classifies each of them into Severity and
Reliability levels. A Risk value is then computed using Fuzzy Logic. The more suspicious
an invariant violation and its corresponding execution path is, the higher it scores in the
Risk scale.
PLATO accepts input from Daikon (step 2) and automates the analysis of the source in step
3. Finally, the PlatoListener is used in step 4 for monitoring JPF’s symbolic execution.
3.4.2. PLATO’s architecture
PLATO consists of components that implement the analysis building block presented in
Section 3. Figure 2 shows the input–output dependencies between these components:
34
Figure 2: PLATO’s processing flowchart
1. The components in orange implement PLATO’s source code analysis, which is based on
the Java compiler to generate the Abstract Syntax Tree (AST) for the AUT. Compiler
methods such as visitIf() and visitMethodInvocation() were overridden, for the analysis of
branch conditions and variable sanitization checks. Finally, information for every single
method invocation, variable assignment and declaration is provided. The following sample
output shows the AST meta-data gathered for the variable sig3 in a class named
Subsystem114:
DECLARE :: 12 :: double :: sig 3 :: 0 :: Main22 :: Subsystem114.java
2. The components in green implement PLATO’s filtering and verification of inferred
dynamic invariants against program states (ℓ,s) at certain code locations that are gathered
from the AUT symbolic execution. The PlatoListener monitors the AUT’s symbolic
execution within JPF.
3. The components in grey implement PLATO’s fuzzy logic system that combines all
information gathered from (1) and (2), and assesses the Risk of execution paths, based on
Mitigation of web threats
35
their position and the analysis findings. Information Gain (Severity) and Cyclomatic
Density (Reliability) computations are performed in this step and the Risk level of each
execution path is calculated by aggregating their output. The Fuzzy Logic system has been
implemented using the jFuzzyLogic library.
3.4.3. Classifying execution paths
Following Oracle’s JAVA API and the related documentation in (Harold, 2006; Gosling,
2014; Java platform, 2015), three categories of Java instructions are proposed for the
classification of execution paths with respect to their Severity and Reliability values. Severity
ranking is based on (i) Input Vector instructions and (ii) potentially exploitable methods (sinks).
Reliability ranking is based on (iii) Control Flow instructions.
3.4.3.1 A Taxonomony of instructions for Severity calculations
About 159 Java methods were reviewed and then grouped into sets depicting danger levels.
These sets are used as features in the Information Gain algorithm to compute the Severity rating
of execution paths. The classified instructions were gathered from NIST’s Software Assurance
Reference Dataset suites (SARD) (Boland, 2012), a set of known security flaws together with
source code test-beds.
Five sets (feature sets) are proposed, corresponding to five danger levels from 1 to 5. The
taxonomy was based on rankings of bugs and vulnerabilities recorded in NIST’s National
Vulnerability Database (NVD, 2015), the U.S. government repository of standards based
vulnerability management data. NVD provides scores that represent the innate characteristics
of each vulnerability using the CVSS scoring system, which is an open and standardized
method for rating IT vulnerabilities.
Thus, each instruction in the taxonomy is assigned to the feature set representing the
appropriate danger level. The correct feature set is inferred based on the CVSS scores in the
NVD repository. This was implemented using the following algorithm:
1. For each instruction, we checked the lowest and highest ratings of NVD vulnerabilities that
use this instruction 2.
2. The characteristics of the identified vulnerabilities are then inputted in the CVSS 3.0
scoring calculator 3, in order to calculate the lowest and highest possible vulnerability
scores.
2 bugs were gathered from the NVD repository:
https://web.nvd.nist.gov/view/vuln/searchadvanced 3 https://www.first.org/cvss/calculator/3.0
36
3. Each instruction was then ranked in the feature set corresponding to the result of previous
step. Instructions with score 7 or above were grouped in Set 5. Instructions with score 6 to
7 in Set 4, those with score 5 to 6 in Set 3, those with score 4 to 5 in Set 2 and those with
score 1 to 4 in Set 1.
Example: The java.lang.Runtime.exec() instruction (Java platform, 2015) is widely-known
to be used in many OS command injection exploits. NVD vulnerabilities recorded 550 using
this instruction have an impact rating ranging from 6.5 up to 10 out of 10. Using the
characteristics of these records, the CVSS scoring calculator outputted a rating of high (7) to
very high (10). This was expected, because exec() is often used to execute code with application
level privileges. Thus, the exec instructions was classified in PLATO’s taxonomy in the very
high (5/5) danger level category.
The types of Java instructions that are included in PLATO’s taxonomy are now presented.
Tables 4 and 5 provide examples for each instruction type. For the full taxonomy, the reader
can access the link at the end of this article.
• Input Vector Methods
Java has numerous methods and classes that accept data from users, streams or files (Java
platform, 2015). Most of them concern byte, character and stream input/output. PLATO takes
into account 69 instructions, one of which is shown in the example shown in Table 5.
Based on (Harold, 2006) and common programming experience, monitoring these
instructions seems to be an adequate way for tracing user input inside Java applications and this
is the reason for using them to create the feature sets for Severity calculations.
java.io.BufferedReader java.io.BufferedInputStream
java.io.ByteArrayInputStream java.io.DataInputStream
java.lang.System javax.servlet.http.
java.io.ObjectInputStream java.io.StringReader
Table 5: Example group - Input Vector Methods taxonomy
• Exploitable Methods (sinks)
Some methods are known to be targeted in exploits. These are the Java instructions that act
as information flow sinks. PLATO takes into account 90 methods gathered from the SARD
sets, which are known to be exploitable. An example is provided in Table 6.
Mitigation of web threats
37
java.lang.Runtime java.net.URLClassLoader
java.lang.System java.sql.Statement
javax.servlet.http.HttpServlet javax.script
java.io. File java.net.Socket
Table 6: Example group - Sink methods taxonomy
3.4.3.2. Instructions for Reliability calculations
Computing the Cyclomatic Density of a source code is tied to the number of execution-
branch statements inside the code. Thus, Reliability calculations take into consideration Java
instructions that affect the program’s control flow.
• Control Flow Instructions
Boolean expressions determine the control flow (Harold, 2006; Fletsmeger, 2010). Such
expressions are found in the statements shown in Figure 3.
All instructions from the mentioned types were gathered from the official Java
documentation (Harold, 2006; Java platform, 2015) and are used for the computations of the
Cyclomatic Density algorithm of Section 3.3.4.
Figure 3: Example types of instructions included in PLATO’s taxonomy
3.4.4. Symbolic execution and PLATO’s JPF listener
PlatoListener is the listener extension implemented in JPF to provide data and
execution paths to PLATO. It monitors all execution transitions, looking for
AssertionViolation instruction invocations. Upon detection of one such invocation,
PlatoListener saves the execution path leading to the assertion violation and compares it
with all other saved execution paths in search of the criterion introduced in Definition 1. For
every such detection, the values of the assertion variables are stored, along with the Java Class
file and references to transitions executed prior to the violated assertion.
38
3.5 Experimental results
Our approach on logical error detection was validated in experiments with both real-world
open-source and “artificially made” applications, namely: (1) real-world software artifacts from
the SIR Object Database (Rothermel, 2006) that are used to support controlled experimentation
on program analysis, (2) a test based on a suite by NIST (Boland, 2012) with logic flaws leading
to exploit manifestation, which is found in a real-world application handing field sensors in a
SCADA system, (3) a real-world controller from NASA’s Lunar Lander, in which a logical
error was injected.
First, we provide a motivating example, in order to introduce our experimentation approach
to validate PLATO’s effectiveness.
3.5.1 An example AUT
We consider an AUT with documented functionality. We will present each step of the
method, as it would be implemented on an example AUT. The steps would be the following:
1. Extracting programmed behaviour in the form of dynamic invariants. Daikon’s
dynamic analysis is used to infer dynamic invariants based on AUT’s executions for use-
case scenarios from the AUT’s business logic documentation. Monitored executions will
have to sufficiently cover all intended functionality, such that the extracted programmed
behavior matches the AUT’s intended behavior (Flegmetser, 2010; Stergiopoulos, 2012).
Example 1: Daikon observes the values of the variables while executing the program and
reports, invariants about that hold true throughout all AUT executions. A dynamic
invariant from Daikon on one of our tests is:
Figure 4: PLATO’s processing flowchart
The dynamic invariant of Fig. 4 shows that, upon invocation of method exec(), the value
of the variable TopLevelChartcount is equal to ’2’.
2. Analyzing source code – Invariant filtering and instrumentation. Invariant rules are
filtered and only those that refer to control locations or to paths with specific error-prone
instructions (high Severity) are kept. Key points in the source code for invariant filtering
are detected by PLATO’s parser that returns the AUT’s AST. Invariants are then parsed
Mitigation of web threats
39
by their method type, variable or class type. Chosen invariants are then instrumented
inside the AUT’s source code as Java assertions.
Example 2. The dynamic invariant in Figure 4 is transformed into a Java assertion and is
then in the source code at the point indicated by Daikon (i.e. the beginning of
Waitforstablerate100000203exec()).
assert this.TopLevelChartcount == 2.0;
3. Monitoring execution paths and program states. JPF executes the instrumented source
code and PlatoListener stores any assertion violations encountered along with the
corresponding execution paths and states backtracked, into structures. In our example, we
recall from Figure 4 that the invariant this.TopLevelChartcount == 2.0 must be true each
time the execution enters the method Wait_for_stable_rate_100000203_exec(). An
example JPF-PlatoListener output is shown in Figure 5.
Figure 5: PLATO’s processing flowchart
Example 3. Let us assume that Fig. 5 shows a path/transition at the beginning of the
Waitforstablerate100000203exec() method. The invariant refers to a variable used in an
IF-Statement and it is therefore chosen for further analysis by PLATO’s filtering system.
It must be true every time the execution enters the rjc.Chart.Waitforstable
rate100000203exec() method. However, it is clear from the memory read in Fig. 5, that
the invariant is violated, because TopLevelChartCount is equal to 1. It is thus clear that
this path violates the corresponding dynamic invariant. An execution path with the same
prefix was found to have TopLevelChartcount = 2. This contradiction is flagged as a
possible logical error.
4. Classifying violations of dynamic invariants.
Example 4. The assertion in Example 2 was found to be both enforced and violated in two
execution paths with the same prefix. The information gain along the path to the invariant
is then computed and the path is classified to Severity and Reliability ranks. These
calculations ultimately assigned a Risk rating of 3.5 out of 5 (High Risk) in this detection.
3.5.2 Experiment 1: real-world airline test from the SIR Object Database
The Software-artifact Infrastructure Repository (SIR) (Rothermel, 2006) is a repository of
software artifacts that supports controlled experimentation with program analysis and software
testing techniques (Do, 2005; Wright, 2010).
40
Our method was tested against a real-world AUT from the SIR repository, which exhibits
the characteristics of a multithreaded activity requiring arbitration. The AUT was a multi-
threaded Java program for an airline to sell tickets.
The logical error. The logical error manifested in this example leads to a race condition
causing the airline application to sell more tickets than the available airplane seats. Each time
the program sells a ticket, it checks if the agents had previously sold all the seats. If yes, the
program stops the processing of additional transactions. Variable StopSales indicates that
all the available tickets were sold and that issuing new tickets should be stopped. The logical
error manifests when StopSales is updated by selling posts and, at the time, more tickets are
sold by the running threads (agents). The AUT’s code is shown in Figures 6 and 7.
Figure 6: SIR AUT example code able to create i threads (agents) which sell tickets
Figure 7: Code: Make sale & check if limit was reached (update ”StopSales”)
Figure 8: Airline sales: No of inferred invariants, chosen assertions and violations
Mitigation of web threats
41
PLATO’s analysis for this test returned the results shown in Figure 8. We now present in
detail the results obtained in the different steps of our workflow:
Step 1-2. There is only one function point to test. Daikon inferred the following invariant
amongst others:
NumOfSeatsSold <= this.MaximumCapacity
Step 3. Dynamic invariants were instrumented in the source code and the software was
symbolically executed in JPF. An assertion violation was detected for the method runBug():
two executions were found where the mentioned invariant was enforced and violated
respectively, thus implying a possible logical error.
Step 4. Our method classified the path in which the invariant assertion was violated with a
Severity = 5 score and a Reliability = 3, thus yielding a total Risk value of 4.5.
3.5.3 Experiment 2: Manipulating remote SCADA RTUs (logical error CWE 840)
This experiment involved a type of logical error included in NIST’s Juliet Test Suite (Boland,
2012) for evaluating static program analyzers and exploit detectors. The Juliet Test Suite is a
collection of over 81.000 synthetic C/C++ and Java programs with a priori known flaws. The
suite’s Java tests contain cases of 112 different CWE vulnerabilities. For our tests, we focus on
the type of vulnerabilities inside the collection that are classified as business logic
vulnerabilities (NIST’s broader CWE-840 group): Vulnerabilities that depend on logical errors
made by the programmer.
Tests selected from the collection are applications which include two methods: A bad()
method in which the logical error manifests and a good() method that implements a safe way
of coding; i.e. a true negative. Since the Juliet Test Case is a synthetic test suite, we mark results
as true positive, if there is an appropriate warning pinpointing the flawed (bad) method in the
code, or false positive if there is the tool flags the non-flawed (good) part of the source code.
This is common practice in such tests (Okun, 2013).
The AUT used in this case study is a real-world application software built on top of the
chosen synthetic test from NIST’s Juliet Test Suite. The developed software is able to handle
Remote Test Unit machinery (RTU) through SCADA systems in a sample critical
infrastructure. Specifically, it can be used to open and close gas pipes according to pressure
measurements and can be used to control pressure valves through an RTU unit and using its
communication protocol.
To communicate and control SCADA systems, the AUT uses a java library for
communicating with SCADA equipment named JAMOD (W. D., 2015). JAMOD applies the
MODBUS protocol to connect software interfaces with RTUs in SCADA systems, using
42
registers that manipulate control circuits (coils, holding registers etc.). The code found in Figure
9 and Figure 10 are sample parts of the AUT’s source code. Using this code, the AUT can be
used with real RTUSs, like the L75 RTU unit. The L75 RTU unit is an RTU that provides
remote control actuation of quarter-turn valves and other rotary devices. It should be noted that,
here, this RTU is not responsible for failure realizations; the flaw manifests in the code
controlling the RTU.
Tests in this experiment were carried out using a MODBUS simulator named MOD RSSIM
(PLCsimulator, 2014) since acquiring a fully operational SCADA control was not feasible. For
our needs, we used the PLCSimulator, originally created to allow testing of Texas Instruments
500 MODBUS RTU serial driver without the need for physical equipment. It supports
MODBUS over TCP, so this makes it ideal for testing purposes.
Figure 9: High-level code able to manipulate a gas shaft in an L75 RTU unit
Software-to-RTU command code – A realistic example. Let Coil 1 be the valve shaft
motion control circuit. A write data command with message ”FF00” either starts or continues
valve movement (if already moving). A write data command with message ”0000” will stop
valve movement. Other data values will return an exception response. Using this, the
hexadecimal version of the MODBUS command able to control and open gas pipes using the
function ”Write Coil” on bus 5 would be: 05, 05, 00, 00, FF, 00, 8D, BE.
The application tested in this case study works as follows: Each time an ”open” (FF00)
command is deployed, the application checks an RTU sensor to see if it can increase pipe
pressure by opening one more valve without reaching the pressure limits. If pressure levels are
too high, the AUT will send a ”close” (0000) command and will not open another pipe. If
pressure remains high, the AUT can send a second ”close” command to reduce pressure. Figure
9 above depicts the high-level instructions of the AUT that are used in the source code to control
the flow of a valve, using the aforementioned commands. When variable stopFlow is false,
an ”open valve” request is sent, otherwise a ”close valve”.
Mitigation of web threats
43
The logical error that manifests in this AUT is a real-world error taken from (Boland, 2012)
and also classified in NIST’s vulnerability database (CWE, 2015).
The logical error: The control system must not utilize more than a specific amount of gas
pipes, otherwise high pressure could cause a rupture, leaks, even a failure to the entire
distribution system. Each time the program sends a request to open a valve, it must check if the
pressure in pipes has reached a maximum or not. If true, it should stop the flow. Variable
”stopFlow” controls this functionality. Inside the AUT’s source code, the logical error
manifests at the part where the boolean variable StopFlow is updated. The software checks a
pressure sensor to see if it should allow further increase in gas pressure or not. This check
should take place each time a valve shaft is ordered to open. Yet, a higher number of pipes than
the allowed maximum can be opened, by bypassing the aforementioned pressure check using
alternate execution routes. This happens due to sensitive information that remains unclear in
objects (i.e. there is a sequence of actions where variable StopFlow is not updated properly).
To better understand this execution deviation, refer to Fig. 10. Due to an erroneous
initialization of variable checked, the control-flow statement at line 6 (if(checked) can be
bypassed if two consecutive ”increase flow” instructions are sent within 3 to 4 seconds (i.e.
before connection resets).
Figure 10: Source code example - Checks imposed to handle pressure limits
We deployed the method presented in Section 5 on this AUT and analyzed execution of the
aforementioned hypothetical scenario for handling pressure in pipes. Figure 11 depicts the
invariant violations and overall statistics of the experiment.
44
Figure 11: CWE 840-RTU: No of inferred invariants, chosen assertions and violations
The results for each step were the following (only output concerning the flaw is presented
due to space limitations):
Step 1. Functionality in this example has two flows currently available:
Exec choice(1), Exec choice(2).
Exec choice(2), Exec choice(1), Exec choice(2).
Step 2. Dynamic analysis of flows yielded 40 dynamic invariants for the selected
functionality. Amongst them, the following invariant refers to a hidden logical error:
Bug.readRegisterPressure():::ENTER
this.checked == false
meaning that, upon entering execution of method readRegisterPressure(), variable checked
should always be FALSE.
Step 3. Dynamic invariants where instrumented inside the source code in their corresponding
points and the software was executed symbolically. An assertion violation was detected for
method readRegisterPressure(): Two executions were found in which the variable checked had
a TRUE and FALSE value respectively, thus violating the invariant assertion from Step 2 and
implying a logical error.
Step 4. Our tool classified this invariant with Severity = 5 (due to source code instructions
manipulating SCADA commands; namely .sendRequest() and WriteSingleRegister) and
Reliability = 2, thus yielding a total Risk value of ˜3.5 (High Risk) for the specific dynamic
invariant and its variable.
Mitigation of web threats
45
3.5.4 Experiment 3: Real-world injection test - NASA’s Apollo Lunar Lander 790
controller
To further validate PLATO’s effectiveness, we also performed a third experiment in which
we injected two faults into a real-world application; namely, NASA’s RJC application, an
application developed to handle NASA’s equipment. A malformed Java object was created that
was initialized with an invalid value. The result of injecting the object in the code was a change
in the AUT execution flow from its intended path to an erroneous one based on the malformed
values, thus causing a logical error in some executions.
Our approach was based on recent results from research on fault injection, which show that
the key issue when injecting software faults is fault representativeness (Natella, 2013): there is
a strong relationship between fault representativeness and fault locations, rather than between
fault representativeness and the various types of faults injected. To pinpoint source code
methods into RJC with relatively high representativeness, we used common software
engineering metrics:
Fault-load representativeness can be achieved by looking at the following metrics: Lines of
Code and Cyclomatic Complexity which represent respectively the number of statements and
the number of paths in a component (Natella, 2013). The Average methods per Class counts
the number of methods defined per type in all Class objects. If this metric scores high, it benefits
these experiments since method invocation paths will be more complex and, therefore, likely
more error-prone. This metric synergizes well with Cyclomatic Complexity in the RJC
experiments. With the above mentioned metrics, we detected methods in RJC that have high
representativeness and then we injected logic errors in them. For our analysis, we used the tool
CodePro Analytix from Google. More specifically, we evaluated the system behavior when one
of its components is faulty and not the behavior of the faulty component itself. We did not
consider additional metrics, as metrics tend to correlate with each other. On the other hand, the
used metrics suffice in order to detect key points in the source code for fault injection (Natella,
2013).
Lines of
Code
Cyclomatic
Complexity
Average
methods
per Type
Rjc.Chart.java 10,48 3,31 29
Rjc.Chart 1.java 13,68 3,31 29
Rjc.Chart 2.java 13,68 3,31 29
Rjc.Reaction Jet Control0.java 99,50 7,50 2
46
Rjc.Reaction Jet Control1.java 85,50 7,50 2
Table 7: Highest metric scores for NASA’s RJC
As we can see in the table above, these five classes have the highest ratings in RJC source
code. ReactionJetControl classes have the highest Lines of Code and Complexity
values. Yet, their average methods per type are significantly low. Also, they have no execution-
defining branch statements inside their code able to diverge the execution of RJC. To this end,
we decided to inject the faulty values of an object controlling an injected control-flow statement
(IF-statement) in the rjc.Chart.Wait_for_stable_ rate_100000203_exec()
method of Chart.java.
Figure 12: RJC Fault injection: No of inferred invariants, chosen assertions and violations
JPF provided the needed method to PLATO in order to cross-check the Daikon-generated
dynamic invariant assertions. As we can see in Fig. 12, 6539 execution paths satisfied the
invariant ”TopLevelChartcount == 2” while 1 injected path violated it.
PLATO went through all the steps of the algorithm: RJC had only one functionality to test
in its source code (Step 1). PLATO detected the dynamic invariants, instrumented the code
with the most critical ones, executed RJC symbolically and detected an assertion violation for
both of the two fault injections (Steps 2 and 3). PLATO’s Fuzzy Logic system classified the
logical error with the following ratings: Severity = 2, Reliability = 1. This is good since this
logical error in the control-flow was injected, meaning that it is irrelevant with the surrounding
code. PLATO detected that this execution path contained no high Severity instructions nor was
”unReliable” enough (very few control flow locations), thus ranked the detection with low
Mitigation of web threats
47
Reliability too. The low overall Risk output produced (Low Risk values around 1.7) suggested
that this detection could be of minor significance; as it indeed was.
A total of 6,240 control flow locations (such as if-statements) were gathered and analyzed
from symbolic execution. Also, 515,854 method invocations and variable Store and Invoke
instructions were processed. The injected paths had 8,064 comparisons. Before injection, all
8,064 paths were found satisfying the rule. As mentioned earlier, after injection, three paths
were found having different states (variable Top-LevelChartcount had different values while
entering and exiting method exec()). Both injected faults were discovered and 850 all possible
deviated execution paths were detected.
3.6 Discussion and comparisons
Comparison with related methodologies and tools
Flaws in source code can be distinguished into many different types and categories. The
CWE vulnerability taxonomy (Martin, 2008), which is co-sponsored by the office of Cyber-
Security and Communications at the U.S. Department of Homeland Security, includes multiple
categories of logic vulnerabilities that can manifest in AUT source code (e.g. CWE-840, CWE-
438 etc.). Our method can be seen as a hybrid. It shares some characteristics with common
static analyzers that look for bugs (such as execution path analysis, AST trees etc.); PLATO
essentially utilizes an entire model checker (JPF) which extends and uses for its own purposes.
Yet, the method is also partially dynamic. It is based on live executions of source code and the
need to have a complete and sound observation of the entire AUT functionality, if it is to
produce dynamic invariants that reflect the AUT’s business logic.
The biggest contribution is differentiation: Most static analysis tools are pattern-specific; for
example FindBugs and other similar tools have shown a rather high detection rate when it
comes to pattern-specific, a priori known flaws (Rutar, 2004), but none is able to detect logical
errors due to erroneous implementation. Since PLATO’s method essentially implements a
higher-level source code analysis, it can be easily combined with such tools that detect flaws at
a purely pattern-specific, source code level. Ideally, PLATO could receive input from these
tools to use as classifiers for its Reliability fuzzy logic function, in an effort to increase the
reliability of its detections. Other approaches (like JPF) allow for the detection of race
conditions, bottlenecks and errors using model checking but still fall short in the detection of
logical errors. For these tools, the best-case scenario is that they may be able to detect some
flaws that may result from logical errors (e.g. race conditions), albeit using a completely
different detection method. Yet, they will miss other flaws or vulnerabilities that stem from
logical errors. Our method detects a superset of this type of errors.
48
3.7 Advantages and Limitations
One of our method’s limitations is the need for input data from live execution of AUTs, while
Daikon infers the likely dynamic invariants. This is an inherent problem in all empirical
methods, since empirical approaches rely on repetitive observations to form rules. Our method
does not model the business logic of AUTs using formal methods, but is rather depended on
the soundness of the likely dynamic invariants provided by Daikon and the various executions
of the AUT. PLATO assumes that it can detect and evaluate the impact of logical errors based
on these properties.
Although empirical methods are often criticized for the lack of sound foundations in software
engineering, it is obvious that, in order for a tool to detect flaws in the logic of applications, it
needs somehow to model knowledge that reflects the intended functionality of each AUT.
Business logic artifacts such as OWL, XBRL etc. fail to adequately describe the full extent of
the practical implementation of an algorithm behind an AUT’s functionality. It is difficult to
use them to automatically parse full sets of logical rules effectively. Also, more often than not,
their use is not widespread amongst developers and even those who do use models like OWL
and XBRL, often fail to follow a common strategy that can be easily analyzed by a parser. This
makes it difficult for a tool to use these kind of artifacts as input to describe the intended
functionality of an AUT. Yet, at the current stage of development, Daikon has repeatedly
proven to be a mature tool for inferring dynamic invariants (Ernst, 2007). Also (provided that
it is executed in the correct manner and that it covers the entire functionality of an AUT)
Daikon’s output does reflect the AUT’s intended functionality, since its dynamic invariants are
properties that were true over the observed executions. PLATO’s results enforce this notion.
The second limitation of this method is the need for targeted AUT functionality analysis. If
PLATO were to examine AUTs of thousands of source code lines in entirety, problems would
arise, mostly due to JPF’s inability to handle large, complex applications and also due to state
explosion. Breaking down large applications to separate functionalities and test each one on
that granularity level can mitigate this.
3.8 Summary
The detection of logical errors in the source code of applications is a tedious and difficult
process, which requires experience and time. Logical errors and, generally, flaws inside the
source code implementation of the business logic of an application can lead to serious
vulnerabilities that can cause failures in important machinery of various critical infrastructures.
Based on the above notion and our tests using PLATO, we drew some significant conclusions:
Mitigation of web threats
49
• PLATO can indeed detect logical errors in applications using reasonable limits in the size
and complexity of AUTs, something no other tool can claim at the time this article was written.
• Results have shown that this method goes beyond logical error detection and can provide
valid detections of other types of flaws that can cause failures in Critical Infrastructures. The
unexpected detection of race conditions in one of our experiments, although it was an
unintended side effect, proved this to be the case. As shown in previous results, limiting a
variables value in an airplane ticket store not only led to a logical error that was essentially a
race condition flaw, but also to a logical vulnerability that could lead the airline to sell more
tickets that its seats.
• Logical errors must be detected using productive reasoning and not inductive because
logical errors can manifest in widely different contexts. For example, a race condition can lead
to a logical vulnerability and is indeed a subtype of logical programming errors, but it can also
lead to other types of errors (null pointer exception, division by zero etc.) or even to no errors
at all. So, detecting known types of errors (like race conditions) and using inductive reasoning
to classify them as logical vulnerabilities is not a reliable classification tactic. Instead, PLATO’s
deductive approach, not only detects various different types of logical errors but can also
provide an insight on the impact of each error, which is context-specific.
• Since results showed that this method can detect a wide variety of errors, it is now possible
to compare different detection methods, output convincing arguments concerning the detection
rates and provide new methods to detect common flaws that might be less resource-intensive.
We consider this for our future work.
50
(this page is intentionally left blank)
Conclusions
51
Chapter 4: Risk Mitigation Strategies for Critical
Infrastructures Based on Graph Centrality
Analysis
4.1 Introduction
Unlike (Inter)dependencies between Critical Infrastructures (CIs) is a key factor for Critical
Infrastructure Protection, since CI dependencies may allow a seemingly isolated on a single CI,
to cascade on multiple CIs. One way to analyze CI dependencies is by using Dependency Risk
Graphs, i.e. graphs in which the nodes represent (modules of) CIs and their directed edges
represent the potential risk that the destination node may suffer due to its dependency from the
source node, in case of a failure being realized at the source CI. In our previous work
(Kotzanikolaou, 2013; Kotzanikolaou, 2013; Kotzanikolaou, 2013; Theocharidou, 2013;
Theocharidou, 2011), we have proposed a risk based methodology that can be used to assess
the cumulative risk of dependency risk paths, i.e. of chains of CI nodes which are
(inter)connected due to their dependencies. Our methodology uses as input the existing risk
assessment results from each CI operator and, based on the 1st-order dependencies between the
CI nodes, it assesses the potential risk values of all the n-order dependency risk chains. By
computing and sorting all the dependency risk paths, the assessor may identify the most critical
dependency chains, by identifying all potential dependency risk paths with a cumulative risk
above a predefined threshold.
Although the identification of the most important dependency chains is an important step
towards efficient risk mitigation, there are still open problems. A simple mitigation strategy
would be to apply security controls at the root node of each critical dependency chain. However,
this may not always be a cost effective strategy, since there are cases of nodes whose effect is
not properly measured. For example, consider the case where a small subset of nodes (not
necessarily being roots of the critical chains) exists that affects a large number of critical
dependency paths. Decreasing the probability of failure in these nodes by selectively applying
to them security controls, may have a greater overall benefit in risk reduction. Another example
is when nodes of high “importance” exist outside the most critical dependency paths; for
example, nodes that may concurrently affect many other nodes which belong to the set of the
most critical paths, or nodes that do affect the overall dependency risk of the entire
structure/graph.
Motivation. Existing risk mitigation methodologies are usually empirical, and they focus on
CIs which are possible initiators of cascading failure and usually belong to specific sectors such
52
as Energy and ICT. By studying actual large-scale failures, it is obvious that these two sectors
often initiate serious cascading effects amongst interdependent CIs (e.g. the famous California
blackout scenario or the recent New York blackout scenario of 2015). However, by only
focusing on some sectors and on potential initiators may not be suitable in every case. The
impact of each dependency should be taken into account together with the position of each CI
within a network of interdependent CIs.
The systematic identification of the most important nodes and the prioritization of nodes for
the application of security controls can therefore be a complex task. The need for a high-level
and efficient risk mitigation technique, which takes into account multiple characteristics of the
interdependent CIs, is a well-known fact (Kotzanikolaou, 2011). An optimal risk mitigation
methodology would allow the detection of the smallest subset of CIs that have the highest effect
on the overall risk reduction on a risk graph, even though the candidate nodes might not initiate
a critical path or may not even belong to the most critical paths. Contribution. Towards this
direction, in this paper we explore the use of graph centrality metrics in Dependency Risk
Graphs, in order to effectively prioritize CIs when applying risk mitigation controls. Our goal
is to design, implement and evaluate the effectiveness of alternative risk mitigation strategies.
Each examined mitigation strategy is implemented algorithmically and it is empirically
evaluated through simulations. The ultimate goal is to identify the minimum subset of CI nodes
within a graph, whose risk treatment (application of security controls) would result in the
maximum risk reduction in the entire graph of interdependent CIs. Our extensions are threefold:
(a) We run various data mining experiments to identify correlations between centrality metrics
and CI nodes that have high impact in a dependency risk graph; (b) We use the optimum
centrality metrics, as identified in the previous step, in order to develop and test various risk
mitigation strategies that maximize risk reduction. Results show that each mitigation strategy
is most effective in achieving specific goals; (c) We take into consideration nodes with a high
number of inbound (sinkholes) and outbound connections and compare risk reduction results.
In order to validate the proposed mitigation strategies, we run experiments on hundreds of
random graphs with randomly selected dependencies. These tests demonstrate the efficiency of
the proposed risk mitigation strategies. All random graphs match criteria found in real-world
CI graphs of interconnected infrastructures. The proposed mitigation strategies can be used as
a proactive tool for the analysis of large-scale dependency infrastructure interconnections and
pinpoint underestimated infrastructures that are critical to the overall risk of a dependency
graph.
Conclusions
53
4.2 Building Blocks
Our methodology will be based on three main building blocks: (i) dependency risk graphs
and multi-risk dependency analysis methodology, in order to model cascading failures; (ii)
graph centrality analysis over dependency risk graphs and (iii) feature selection techniques in
order to evaluate the effect of various centrality metrics in risk mitigation. We briefly describe
these building blocks.
4.2.1. Multi-risk dependency analysis methodology
Dependency can be defined as “the one-directional reliance of an asset, system, network, or
collection thereof –within or across sectors– on an input, interaction, or other requirement from
other sources in order to function properly” (U.S. Department of Homeland Security, 2013).
Our work extends the dependency risk methodology of [1, 2] used for the analysis of multi-
order cascading failures.
4.2.1.1. First-order dependency risk
In (Kotzanikolaou, 2013; Kotzanikolaou, 2013) initially the 1st-order dependencies between
CIs, as identified and are then extended to model the n-order relations. Each dependency from
a node CIi to a node CIj has been assigned1 an impact value, denoted as Ii,j and the likelihood
Li,j of a disruption being realized. The product of the last two values defines the dependency
risk Ri,j caused to the infrastructure CIj due to its dependency on CIi. Dependencies are
visualized through graphs G = (N, E), where N is the set of nodes (or infrastructures or
components) and E is the set of edges (or dependencies). The graph is directional and the
destination CI is receiving a risk from the source CI due to its dependency.
4.2.1.2. Extending to n-order dependency risk
Let ℂ𝕀 = (CI1, ... CIm) be the set of all the examined infrastructures. The algorithm examines
each CI as a potential root of a cascading effect. Let CIY0 denote a critical infrastructure
examined as the root of a dependency chain and CIY0 → CIY1 → ... → CIYn denote a chain of
length n. Then the algorithm computes, for each examined chain, the cumulative Dependency
Risk of the n-order dependency chain as:
𝐷𝑅𝑌0,…,𝑌𝑛 = ∑ 𝑅𝑌0,…,𝑌𝑖
𝑛
𝑖=1
≡ ∑ (∏ 𝐿𝑌𝑗−1,𝑌𝑗
𝑖
𝑗=1
) ∗ 𝐼𝑌𝑖−1,𝑌𝑖
𝑛
𝑖=1
Informally, equation 1 computes the dependency risk contributed by each affected node in
the chain, due to a failure realized in the source node. The computation of the risk is based on
54
a risk matrix that combines the likelihood 105 and the incoming impact values of each vertex
in the chain. Interested readers are referred to (Kotzanikolaou, 2013) for a detailed analysis.
4.2.2. Graph centrality metrics
Graph centrality analysis attempts to quantify the position of a node in relation to the other
nodes in a graph. In other words, graph centrality metrics 110 are used to estimate the relative
“importance” of a node within a graph. Various centrality metrics are used, each of them
measuring a different characteristic:
Degree centrality measures the number of edges attached to each node. Given a node u,
degree centrality is defined as: Cd(u) = deg(u), where deg(u) is the total number of its
outgoing and ingoing edges.
Closeness centrality quantifies the intuitive “central” or “peripheral” placement of a node
in a two dimensional region, based on geodesic distances. It is defined as: 𝐶𝐶(u) =
∑ 𝛿 (𝑢, 𝑣)∀v∈V (G) , where δ(u,v) is defined as the average shortest path between the
examined node u and any other node in the graph.
Betweenness centrality measures the number of paths a node participates in, and is defined
as: 𝐶𝑏(u) = ∑ 𝛿𝑖𝑗(𝑢)u≠ i ≠ j ∈ V , where 𝛿𝑖𝑗(u) = 𝜎𝑖𝑗(𝑢)
𝜎𝑖𝑗. Here, σij(u) denotes the
number of geodesic distances rom i to j where node u is present and σij the number of
geodesic distances from i to j.
Bonacich (Eigenvector) centrality (Bonacich, 1987) attempts to measure the influence of
a node in a network as: 𝑐𝑖(𝛼, 𝛽) = ∑ (𝑎 − 𝛽𝑐𝑖)𝑅𝑖𝑗 𝑗 , where α is a scaling j factor, β reflects
the extent to which centrality is weighted, R is the node adjacency matrix, I is the identity
matrix and l is a matrix of ones. An adjacency matrix is a N×N matrix with values of 1 if
there is an edge between two nodes and 0 otherwise.
Eccentricity centrality is similar to closeness centrality. Essentially, it is the greatest
distance in all shortest-paths between u and any other vertex (geodesic distance).
4.2.3. Feature selection
Feature selection (also called variable selection or attribute selection) is used in data mining
for the automatic selection of attributes in a data set that are 6 most relevant to a predictive
modeling problem. Feature selection is critical for the effective analysis of characteristics in
datasets, since data-sets frequently contain “redundant” information that is not necessary to
Conclusions
55
build a predictive model (Microsoft, 2015). It increases both effectiveness and efficiency, since
it removes non-informative terms according to corpus statistics (Yang, 1997).
In our case, we will explore centrality metrics as potential features in a dependency risk
graph. Our predictive model is to detect which features (i.e. centrality metrics) are able to
characterize nodes that mostly affect the cumulative risk of dependency paths. This model will
identify which centrality metrics are the most useful (i.e. provide the most information) about
whether a CI node greatly affects the overall risk on a graph or not. The various centrality types
are ranked lower if they are common in both the so-called positive set (nodes that have a high
affect on the risk) and the negative set (nodes with low impact on risk paths). The features are
ranked higher, if they are effective discriminators for a class. Our feature selection tests will
use two well-known selection techniques, namely Information Gain and Gain Ratio.
Information Gain (Abramson, 1963) is a statistical measure that has been successfully
applied for feature selection in information retrieval. Authors in (Glover, 2001; Ugurel, 2002)
used this algorithm for feature selection before training a binary classifier. This technique will
be used to classify high centrality metrics in nodes, in an effort to distinguish, which centrality
metrics tend to appear often in nodes that greatly affect the cumulative risk of graphs. Gain
Ratio is a variation of the Information Gain algorithm. The difference is that the Information
Gain measure prefers to select attributes having a large number of values (Karegowda, 2010).
Gain Ratio overcomes this bias by taking into consideration the statistical appearance of a
feature in both negatives and positive sets (i.e. if a specific Centrality metric is detected in
dangerous CI nodes scarcely but is never found in safe nodes, it will have a rather low
information gain rank but a very high gain ratio one). We use this technique to pinpoint
(combinations of) Centrality metrics that might scarcely appear in graphs, yet significantly
affect the cumulative risk in paths when present.
4.3 Exploring Centrality Metrics in Dependency Risk Graphs
We will explore the effect of centrality metrics on the “importance” of nodes in dependency
risk graphs. Recall that in such graphs the edges denote directed risk relations between nodes,
and not topological connections. Intuitively, nodes with high centrality measures are expected
to have a high effect in the overall dependency risk. The question that needs to be answered is
which centrality metrics have the higher effect in Risk propagation. Then, the nodes that are
exhibiting the highest values of these centrality metrics will be good candidates for the
implementation of risk mitigation controls. An algorithm capable of identifying these nodes
will be able to design a cost-effective risk mitigation strategy.
56
4.3.1. Analysis of centrality metrics
We will examine each centrality metric in dependency risk graphs, in order to analyze its
effect to the overall risk.
4.3.1.1. Degree centrality
A node with high degree centrality is a node with many dependencies. Since the edges in risk
graphs are directional, the degree centrality should be examined for two different cases (Luiijf,
2009):
Inbound degree centrality, i.e. the number of edges ending to a node. A high inbound degree
centrality would indicate nodes known as cascading resulting nodes.
Outbound degree centrality, i.e. the number of edges starting from a node. This is an
indication of a cascading initiating node.
Nodes with high inbound degree centrality in a risk graph are natural “sink-hole” points of
incoming dependency risk. Nodes with high outbound degree centrality seem to be suitable
nodes to examine, when prioritizing mitigation controls. Indeed, if proper mitigation controls
are applied to such nodes, then multiple cumulative dependency risk chains will simultaneously
be reduced.
Such a strategy could be a much more cost-effective alternative, from a mitigation strategy
aiming at applying controls to particular high-risk edges or high-risk paths. Obviously, it is not
certain that applying one or more security controls to a node with high outbound degree
centrality, will positively affect many (or all) outgoing dependencies chains using this node.
However, a mitigation strategy would benefit if it were to initially examine such possible
security controls.
4.3.1.2. Closeness centrality
Nodes with high closeness centrality are nodes that have “short” average distances from most
nodes in a graph. In the case of a dependency risk graph, nodes with high closeness are nodes
that tend to be part of many dependency chains; sometimes might even initiate them. Since
cascading effects tend to affect relative short chains (empirical evidence indicates that cascades
rarely propagate deeply (Eeten, 2011), intuitively, nodes with high closeness centrality will
have a bigger effect in the overall risk of the dependency chains, than nodes with low closeness
centrality. To formalize this, consider equation 1 used to compute the cumulative risk of a
dependency chain: the closer a node is to the initiator of a cascading event, the more effect will
have on the cumulative dependency risk since the likelihood of its outgoing dependency will
affects all the partial risk values of the following dependencies (edges).
Conclusions
57
A more effective way to exploit closeness centrality in mitigation decisions is to compute the
closeness of every node with respect to the subset of the most “important” initiator nodes.
Regardless of the underlying methodology, the risk assessors will already have an a priori
knowledge / intuition of the most important nodes for cascading failure scenarios. For example,
empirical results show that energy and ICT nodes are the most common cascade initiators
(Eeten, 2011). In addition, nodes with high outgoing degree centrality will probably be nodes
that participate in multiple dependency risk chains. Thus it is possible to first identify the subset
of the most important nodes for cascading failures and then compute the closeness of all others
in relation with this subset of nodes, as a secondary criterion for mitigation prioritization.
4.3.1.3. Eccentricity centrality
Similar to closeness centrality, this is a measure of the centrality of a node in a graph which
is based on having a small maximum distance from a node to every other reachable node. Here,
small maximum distances are the greatest distances detected in all shortest-paths between v and
any other vertex (geodesic distance). If the eccentricity of a CI node is high, then all other CI
nodes are in proximity.
4.3.1.4. Betweenness centrality
In a dependency risk graph, a node with high Betweenness centrality will lie on a high
proportion of dependency risk paths. This means that even though such nodes may not be
initiating nodes of a cascading failure (high outbound centrality) or may not belong to a path
with high cumulative dependency risk, they tend to “contribute” to multiple risk paths and thus
play an important role to the overall risk. Applying mitigation measures to such nodes (in the
form of security controls) will probably decrease the dependency risk of multiple chains
simultaneously.
Comparing closeness centrality with Betweenness centrality, it seems that closeness should
precede Betweenness as a mitigation criterion. Although nodes that are between multiple paths
will eventually affect multiple chains, it is possible that a node lies in multiple paths but it tends
to be at the end of the chain and practically not affecting the cumulative dependency risk chain
(recall that nodes with high-order dependency are rarely affected).
4.3.1.5. Bonacich (Eigenvector) centrality
A node with high Bonacich (Bonacich, 1987) (eigenvector) centrality is a node that is
adjacent to nodes with very high or very low influence, depending on whether variable β is β >
0 or β < 0. In a risk dependency graph, we are interested in nodes with high eigenvector
centrality where β > 0, since these nodes are connected to other nodes which also have high
connectivity. This is an interesting measure for CI risk graphs, as such nodes not only can cause
cascading failures to more nodes, but they can cause multiple cascade chains of high risk. On
the contrary, a less connected node means that it shares fewer dependencies with other nodes.
58
Only specific nodes in the graph affect it. This means that applying mitigation measures to such
a node may not affect significantly the overall risk. On the contrary, if one applies mitigation
controls to the node with high Eigenvector centrality (when β > 0), this means that the most
powerful (or critical) is modified and this, in turn, affects several other important nodes.
4.3.2. Applying feature selection to classify centrality metrics
In order to validate our intuitive analysis of centrality metrics in dependency risk graphs,
described in section 4.3.1, we will use a data mining technique named feature selection to
empirically study possible relations between the risk level of nodes in dependency paths (as
assessed using the aforementioned methodology from (Kotzanikolaou, 2011)) and the various
centrality metrics for nodes. This experiment will show how often nodes concurrently appear
in; (a) the set of the most critical paths (i.e., the paths with the highest cumulative dependency
risk value) and (b) to sets of nodes with the highest centrality values in a dependency risk graph.
For our experiments, we simulated graphs that were randomized based on restrictions
proposed in recent empirical studies (Luiijf, 2009; Eeten, 2011), in order to resemble CI
dependencies based on real data. In particular:
Occasional tight coupling. Some node relationships in the dependency risk graph are
selected to have high impact values. To achieve this, in our experiments randomization
applies random risk values with relatively high lower and upper likelihood and impact
bounds.
Interactive complexity. In reality, dependencies between CIs have high interactive
complexity. Informally, this implies that we cannot foresee all the ways “things can go
wrong”. In our experiments, we constructed random graphs of 50 nodes with high
complexity; the critical paths up to 4th-order dependencies have possible combinations
ranging from 230,300 to 2,118,760 chains.
Number of dependencies. We randomly simulated CI nodes having from 1 to 7 connections
(dependencies).
Path length. Critical paths of length 3 to 4 hops are used.
Number of initiators. More than 60% of the nodes in the graph act as initiators.
Connectivity of initiators. In our experiments, the initiators tend to have higher connectivity
than the other nodes.
In our experiments we used the Weka4 tool (version 3.6). All experiments were conducted
using the available implementation of the Information Gain and Gain Ratio Feature selection
algorithms in Weka4, using the program’s “Attribute Selection Ranker” to rank output results.
4 http://www.cs.waikato.ac.nz/ml/weka/
Conclusions
59
4.3.2.1. Results of feature selection for centrality metrics
We analyzed a dataset of 32,950 nodes extracted from about 700 graphs containing about
774,015,270 paths. Dependency graphs were created with the aforementioned restrictions.
Feature selection algorithms were used to detect correlations between high centrality metrics
and CI nodes that greatly affect the overall risk.
Weka’s output rankings provide insight on how high centrality features characterize
dangerous nodes that affect the risk in top critical paths. Detailed output values for each test are
presented in Table 8 and 9. Table 1 depicts the output from the Information Gain algorithm
whereas Table 9 provides the Gain Ratio output:
Table 8. Weka’s output ranking using the Information Gain algorithm
Table 9. Weka’s output ranking using the Gain Ratio algorithm
Tests are divided into two sets: Inbound tests and Outbound tests. The difference between
the two is that Inbound tests use Inbound Degree Centrality measures whereas Outbound tests
compute everything using Outbound Degree Centrality measures. Results from Weka provided
valuable information as to how nodes with high centrality values affect dependency risk graphs:
1. The amount of critical paths that had at least one node with high centrality measurements,
is very high: An average of 74,4% of the most critical path in all graphs contained a node
with at least one high centrality metric. This percentage seemed to increase dramatically
to 99.9% for the top 10% of most critical paths, which leads to the conclusion that, the
top 10% of paths frequently pass through the same nodes as the top 1% of paths. If we
analyze even larger sample sets (more than 10% of critical paths) almost all nodes with
high centrality are part of some critical path.
60
2. Gain ratio showed that CI nodes with high centrality measures in all types of centralities
rarely appear in graphs but, when they do appear, they definitely affect the cumulative
risk of critical paths. CI nodes that have high measures in all centrality metrics are of high
importance.
3. Nodes with high Closeness centrality seem to be the best candidates for detecting
important CI nodes. Yet, results clearly show that all centrality types seem to be useful
indicators of important CI nodes (information gain is above 20% for all features). Thus,
all centrality metrics must be taken into account in each graph.
4. Nodes with high Closeness and/or Degree centrality, both outbound and inbound
(sinkholes), tend to mitigate high amounts of risk. An average of 49,6% of all dangerous
nodes tested had a high Closeness centrality and an average of 45% had a high Degree
centrality value.
5. Risk mitigation is highly depended on the graph itself. For example, in graphs that contain
nodes acting like single points of failure, high centrality values are excellent indicators of
important CI nodes. If however we examine graphs in which, for example, all nodes have
equal degree, the identification of nodes for risk mitigation is often less successful.
Luckily, CI dependency nodes tend to have specific structural characteristics.
4.4 An Algorithm for Efficient Risk Mitigation Strategy
We have concluded from the previous experiment that, even if nodes with high centrality are
only a fraction of the nodes in the most critical risk paths, yet, they do affect all critical paths
greatly. Also, feature analysis results suggest that all Centrality measures can play an important
role, depending on the graph analyzed. Thus, it seems essential for a risk mitigation strategy
not only to take into consideration which Centrality measures provide the best results (like high
Closeness measures) but also to consider the diversities of each graph and adapt accordingly
by choosing the correct combination of Centrality metrics to pinpoint nodes for applying risk
mitigation controls.
We incorporate these results from the previous feature selection analysis Section 4.3.2), in
order to design a node selection algorithm for efficient risk mitigation. The proposed algorithm
is heuristic and recursively chooses the best set of nodes, by limiting its selection options, until
it reaches a set of selected nodes of a predefined size.
4.4.1. Modeling feature selection results to algorithmic restrictions
Recall from Section 4.3.2 that the two most important conclusions support that: (i) Nodes
with high centrality measures in all types of centrality metrics always affect the total risk of a
graph; (ii) Nodes with high Closeness and high Betweenness centrality seem to have a greater
Conclusions
61
impact than the other metrics; (iii) If a node concurrently has high centrality values for more
centrality metrics, it tends to have a higher affect on the risk graph. To model these observations
into a dynamic node selection algorithm for risk mitigation, we implemented the following
steps:
1. Model Definition: Define the subsets of nodes which exhibit the highest centrality values
for each centrality metric, as well as their Information Gain over the entire node set.
2. Setup: Pre-compute all centrality measures for each node in a graph. Using these centrality
measures, generate all high centrality subsets. These subsets will act as features for feature
selection. In addition, pre-compute all dependency risk paths of the graph and sort them
according to their weight (total cumulative risk value of each path).
3. Use decision trees and Information Gain for node selection: Define the Information Gain
method and its use in splitting node sets.
4. Traverse a decision tree using Information Gain splits: Traverse a decision tree to select
the optimal subset of nodes for applying risk mitigation controls.
4.4.1.1. Model Definition
Let CA denote the subset of the top X% of nodes with the highest centrality measurements
from all centrality sets; CE denote the subset of the top X% of nodes with the highest
Eccentricity, CC denote the subset of the top X% of nodes with the highest Closeness centrality;
CI denote the subset of the top 380 X% of nodes with the highest Inbound Degree centrality
(sinkholes); CB denote the subset of the top X% of nodes with the highest Betweenness
centrality and finally CEg denote the subset of the top X% of nodes with the highest EigenVector
centrality.
Let N denote the set of all nodes in the risk graph. Let D ⊂ N denote the subset of nodes that
participate in the top 20 most critical risk paths of the graph. Finally, let IGx(N) denote the
information gain when choosing subset Cx as a feature of the node set N. For example, IGB(N)
denotes the information gain when using high Betweenness centrality as a feature of set N.
Information Gain values are calculated for each centrality feature subset separately. The high
centrality subset Cx with the highest Information Gain over N, is considered to affect at most
the risk graph, in comparison with the other high centrality subsets.
Example: In a Dependency Risk Graph, there exist nodes that act as initiators of cascading
events in the three most critical paths (i.e., critical paths with the highest cumulative
dependency risk value). Yet nodes exist that, even though they have smaller influence on the
top three most critical paths, they affect the overall risk of the entire Dependency Risk Graph
more than the aforementioned initiators. Our risk mitigation algorithm will be able to suggest
62
these nodes over the three initiator nodes, using the Information Gain output from all high
centrality node subsets.
4.4.1.2. Setup
The centrality measures of each node are pre-computed, along with all the dependency risk
paths of the graph. For each Centrality measure, the algorithm uses the precomputed centrality
values and creates subsets CE,CC,CI,CB and CEg. Then, the algorithm computes all risk paths
in the graph and sorts them according their total accumulated risk. All output will be used as
input to the risk mitigation algorithm to choose the most effective nodes for risk mitigation
using countermeasures.
4.4.1.3. Using decision trees and Information Gain for node selection
A decision can be made (and, practically, a decision tree can be traversed) by creating a split
over the head set N of CI nodes and create smaller subsets (graph leaves). In a Dependency
Risk Graph, the algorithm chooses the specific centrality node subset with the highest impact
on the graph. Then, it outputs a new subset that is the intersection of the chosen subset with the
original set. This process is repeated on each derived subset in a recursive manner called
recursive partitioning. The recursion is completed when the Information Gain calculation at a
specific point in the decision tree produces the same Information Gain values for the target
subset of nodes, or when splitting no longer adds information to the predictions.
The algorithm in 4.4.1.3 is used to define a sequence of high centrality subsets that act as
features. The risk mitigation algorithm investigates this sequence to rapidly narrow down the
group of CI nodes with the highest impact in the Dependency Risk Graph. This feature selection
increases both effectiveness and efficiency, since it removes non-informative CI nodes
according to corpus statistics (Yang, 1997).
Conclusions
63
4.4.1.4. Example: Traversing a decision tree using Information Gain splits
The above algorithm aims to choose nodes with the highest impact on a Dependency Risk
Graph. The above risk mitigation strategy can be simplified into the following sentence: Always
choose the subset of nodes with the highest Information Gain from each step in the decision-
64
tree. Let us assume to have a Dependency Risk Graph with all centrality measures calculated
and all high centrality subsets defined. Thus the algorithm 4.1.3 would run as follows:
1. Step 1: We need to find which centrality feature produces the most Information Gain
against the root set D ⊂ N of nodes. The information gain is calculated for all four high
Centrality subsets (features):
IGC(D) = 0.246 // Gain of Closeness subset over D
IGB(D) = 0.029 // Gain of Betweeness subset over D
IGS(D) = 0.051 // Gain of Sinkholes subset over D
IGEg(D) = 0.151 // Gain of Eigenvector subset over D
IGE(D) = 0.048 // Gain of Eccentricity subset over D
The Closeness high centrality subset provides the highest Information Gain; therefore it is
used as the decision attribute in the root set D. From here on, D′ is the subset D∩CC of nodes
and N becomes the N ∩CC subset.
2. Step 2: The next question is “what feature should be tested at the second branch?” Since
we have used CC at the root, we only decide on the remaining four centrality subsets:
Betweenness, Sinkholes, Eigenvector and Eccentricity.
IGB(D) = 0.246
IGEg(D) = 0.029
IGS(D) = 0.051
IGE(D) = 0.000
In this step, the Betweenness subset shows the highest Information Gain; therefore, it is
used as the decision node. From here on D′ is the subset D ∩ CB of nodes and N
′ becomes
the N ∩ CB subset.
3. Step 3 to Step N: This process goes on until all nodes are classified and D remains with
three nodes, or if the algorithm runs out of Centrality attributes.
4.5 Validation of the proposed algorithm
We empirically study the efficiency of the risk mitigation algorithm presented in Section 4.4.
We run 2000 random experiments; in each experiment the graph was produced based on the
limitations described in Section 4.3.2 and each graph contained up to 50 nodes.
Conclusions
65
4.5.1. Efficiency analysis of the proposed mitigation strategy
We know now from previous results that centrality measures can indeed characterize
important CI nodes that greatly affect the overall risk in a dependency risk graph. To evaluate
the effectiveness of the proposed risk mitigation strategy we need to emulate the
implementation of mitigation controls in the nodes identified as important (dangerous) nodes
by the algorithm presented in Section 4.4. In practice, examples of controls can be the repair
prioritization of nodes (which node to send a repair crew first), the increase of asset redundancy
on a node, or any other control that may reduce the likelihood or the consequence of a failure.
In our experiments, we emulate the implementation of mitigation controls to a node CIi, by
reducing the values Li,j that define the likelihood to cascade a failure experienced at node CIi
to another node CIj, ∀ node CIj that depends on node CIi. All likelihood values Li,j of all the
dependencies that originate from CIi were reduced by 20%. To measure the result of applying
risk mitigation controls on each selected subset of nodes, we calculated the dependency risk
values in the same graph, before and after the implementation of risk mitigation and we
calculated the risk reduction achieved in each case.
We measured the efficiency of the proposed risk mitigation algorithm by calculating the
percentage of risk reduction in graphs using the aforementioned emulation of mitigation
controls. The effect on three risk metrics was examined using the proposed algorithm 4.4.1.3
from Section 4.4: (i) risk reduction achieved in the most critical path, (ii) risk reduction in the
total sum of the risk of the top 20 paths of the highest cumulative dependency risk and (iii) risk
reduction in the total sum of the entire risk from all graph paths. Only acyclic paths were taken
into consideration to reduce overlapping: For example, if a high-risk path was cyclic, its acyclic
part would most likely appear as a high risk path too. Thus, we avoided calculating cyclic paths
in our experiments.
Mitigation controls were implemented on three (3) out of the fifty (50) CI nodes in each
experiment (6% of the nodes in the entire graph).
66
4.5.1.1. Effect of the algorithm in the most critical path
Figure 13. Risk reduction in the most critical path
Using the proposed algorithm from Section 4.4 to select CI nodes for implementing
mitigation controls, we achieved a average risk reduction of 12.1%. The highest risk reduction
achieved in all experiments was 43.7%; almost half the risk of the most critical path was
mitigated. The risk mitigation achieved was the highest achieved in all different mitigation
strategies tested. A more thorough comparison of the results can be found in the following
section 4.5.2.
As it is shown in Figure 13 in about 25% of the times (498 out of the 2000 tests) there is no
risk reduction to the most critical path. This however is not a primary concern, since our goal
is to reduce the overall risk of the dependency risk graph and not necessarily on nodes of the
most critical path that are easy to identify. We note however that when mitigate controls were
applied on nodes belonging to the most critical path, the algorithm achieved better risk
reduction results than alternative well-known mitigation strategies, such as the implementation
of mitigation controls on the initiator of the most critical path.
4.5.1.2. Effect of the algorithm in the top 20 risk paths
In Figure 14, we see that the highest risk reduction achieved in the sum of all the risk values
derived from the top 20 critical paths, was 37.5%. The average risk reduction achieved in graphs
using the proposed algorithm was 9.8%. In addition, we see that, in 2000 experiments, only 12
graphs had a structure that prohibited our algorithm from achieving any risk reduction in the
Conclusions
67
top 20 paths; 99.6% of the algorithm’s applications achieved some risk mitigation and the
47.3% of applications achieved high-risk reduction (10% or more).
Figure 14. Risk reduction in the top 20 most critical paths
These are interesting results: The top 20 most critical paths are comprised of 1000 nodes in
each experiment. Even though paths share CI nodes, being able to choose only 3 out of 50 nodes
for risk mitigation in 20 paths is far more complex than choosing the correct nodes only in the
most critical path comprised of 5 nodes. Again, the proposed algorithm had the best
performance out of the three different mitigation strategies tested.
In addition, by monitoring the algorithm’s execution and CI node choices, we validated some
preliminary results from Feature Analysis: The risk mitigation algorithm was mostly choosing
CI nodes with high centrality measures in all centrality metrics (i.e. the intersection of all high
centrality sets for all centrality metrics). Besides aggregating all centrality metrics, the second
most frequent combination of centrality types was nodes with high Degree centrality (e.g.
sinkholes) and Closeness centrality.
68
4.5.1.3. Effect of the algorithm in the entire dependency risk graph
Figure 15. Risk reduction in the complete set of paths
Last but not least, we analyzed the efficiency of the proposed algorithm when trying to
reduce risk in the entire risk graph; not only on the top 20 paths. Here, we tested risk mitigation
using the proposed node selection algorithm on the sum of all paths in dependency graphs. The
results from 2000 experiments are shown in Figure 15. We see that the highest risk reduction
achieved was 12.2%, with an average risk reduction of 7.5% in all experiments. Interestingly,
in 86.4% of the tests, the risk reduction achieved was between 5% and 10% with an average of
around 8.7%. With such low dispersion of results, statistical probability suggests that the
proposed algorithm’s output shows strong signs of uniformity and homogeneity. Again, the
proposed algorithm had the best performance out of the three different mitigation strategies
tested.
4.5.2. Comparing results with alternative mitigation strategies
After presenting results of the proposed algorithm, we will now compare our proposed risk
mitigation algorithm with alternate mitigation mechanisms to evaluate its performance. Two
alternate mitigation mechanisms for choosing CI nodes were used, both well-known and
frequently proposed scenarios.
4.5.2.1. Using initiators from the most critical paths
The first strategy is based on the idea that initiators are critical points of failure in dependency
risk chains. The first alternate mitigation strategy chooses to implement mitigation controls on
CI nodes that are the initiators of critical dependency paths. Thus, in our experiments, three (3)
Conclusions
69
out of the fifty (50) CI nodes are chosen (6% of the nodes in the entire graph) that act as
initiators of the corresponding top 3 most critical paths.
4.5.2.2. Using nodes with high Degree Centrality (sinkholes and outbound)
The second strategy focuses on the common idea amongst auditors that CI nodes with high
inbound and outbound Degree Centrality (i.e. nodes with a high amount of incoming and/or
outgoing connections) affect dependency risk paths a lot. For this reason, the second mitigation
strategy chooses nodes the top 6% of nodes with the highest inbound and outbound Degree
Centrality measures to apply mitigation controls.
4.5.2.3. Comparison results
Table 10. Comparison of results from all mitigation strategies
Strategies Information Gain Top Initiators Top Sinkholes
Risk Metrics
Most critical path 43.7% (max)
12.1% (avg) 38.4% (max)
11.8% (avg) 34.5% (max)
10.3% (avg)
Top 20 critical paths 37.5% (max) 9.8% (avg)
28.7% (max) 10.0% (avg)
29.8% (max)
7.3% (avg)
Entire graph 12.2% (max)
7.5% (avg) 10.1% (max)
5.3% (avg) 10.8% (max)
6.7% (avg)
Table 10 shows the results from all mitigation strategies. Output marked in red exhibits the
highest values achieved. Clearly, in our tests, the proposed decision making algorithm using
Information Gain had a clear advantage over the rest as far as pure numbers are concerned.
Using the top initiators mechanism scored a higher average in reducing risk at the top 20 most
critical paths, albeit only marginally.
4.5.3. Implementation details
We briefly describe the technical aspects of the executed tests; platforms, technologies and
algorithm implementations. All the tests were performed on an Intel Core i-7 processor
@2.7GHz with 4 cores and 16 GB of RAM. In order to test the proposed risk mitigation
strategies, we developed an automatic Dependency Risk Graph generator in Java. The
application is able to randomly create risk graphs and, then, run the mitigation algorithms on
the developed models. Finally, it provides information on the percentage of risk reduction
achieved using each of the mitigation mechanisms.
We selected Neo4J as a technology to implement our dependency analysis modeling tool,
due to its adaptability, scalability and efficiency (Juili, 2013; Shao, 2012). Neo4J builds upon
the property graph model; nodes may have various labels and each label can serve as an
informational entity. The nodes are connected via directed, typed relationships. Both nodes and
relationships hold arbitrary properties (key-value pairs). Using the Neo4J technology, the
70
developed framework can represent complex graphs of thousands of dependent CIs through a
weighted, directed graph. The developed application is stand-alone: It builds a random
Dependency Risk Graph using the aforementioned restrictions to imitate real-world CI
dependencies. It then pre-calculates all necessary graph data and executes the selected risk
mitigation algorithm. When finished, it outputs a percentage of the risk reduction achieved in
the current experiment/graph.
4.6. Conclusions
In this paper, we extended our previous method on dependency risk graph analysis, using
graph centrality measures as additional criteria for the evaluation of alternative risk mitigation
strategies. We used feature selection algorithms and confirmed that the most critical paths in
dependency risk graphs tend to involve nodes with high centrality measures. Since various
centrality metrics exist, we performed multiple tests in order to define which measures and
combinations are the most Our results showed that aggregating all centrality sets to identify
nodes with high overall centrality provides an optimal mitigation strategy. However, by
comparing the different centrality metrics, we showed that nodes with high Closeness and high
Degree centrality measures seem to have the highest impact on the overall risk of a graph.
Still, simply choosing nodes from these sets is not always a viable choice, as there exist
dependency graphs where no nodes exist in specific high centrality sets or there may be
contextual reasons that inhibit the application of controls in these nodes. For this reason, we
developed a risk mitigation algorithm that uses a decision tree built from subsets of nodes with
high centrality measures for each centrality metric. The Information Gain algorithm is used to
traverse the tree and choose subsets of high centrality nodes that best characterize each graph
under test. Thus, the algorithm is able to dynamically pinpoint the optimal subset of nodes with
high centrality features, for overall risk mitigation.
4.6.1. Advantages and Limitations of the proposed mitigation method
Although the proposed mitigation algorithm achieved the highest maximum risk mitigation
when compared with well-known mitigation strategies, our experiments showed that there are
cases where the algorithm will not select nodes belonging to the most critical path set. This can
be considered as a potential limitation, and at the same time, a desired characteristic. On one
hand, the proposed algorithm will not always reduce the risk of the most critical path. Note
however that if this is required, it can be trivially achieved by selecting the initiator of the most
critical path. On the other hand, the proposed algorithm outperforms all its competitors, since
it searches for nodes that have a great effect at the overall risk and not for a subset of critical
Conclusions
71
paths. This is the main advantage of the proposed algorithm; it can be used to identify CI nodes,
which may not appear in the top critical path(s), but they greatly affect the overall risk of a
dependency graph. While the exhaustive computation of the complete set of dependency risk
paths may provide useful information and reveal underestimated CI nodes, assessors may also
use the algorithm to examine particularly complex scenarios were manually selecting nodes for
implementing mitigation controls might be very time consuming and/or impractical.
A limitation of this approach is the need for input data from risk assessments performed at
the each CIs. This is an inherent problem of all empirical risk approaches, since they analyze
dependencies based on previous incidents. Sector coordinators or regulators may collect data
concerning a specific sector (such as ICT or energy). National CIP authorities or CERTs may
be able to collect such information.
72
Chapter 5: Time-based Critical Infrastructure
Dependency Analysis for Large-Scale and Cross-
Sectoral Failures
5.1 Introduction
Most Critical Infrastructures (CIs) can be modeled as cyber-physical systems whose cyber
components control their underlying physical components. CIs are inherently complex systems
since they integrate heterogeneous platforms, proprietary systems, protocols and open
communication networks. In addition, CIs are usually interconnected and interdependent with
other CIs that may belong to different sectors (such as energy, ICT or transportation).
According to (Rinaldi, 2001) CIs may have physical, informational or logical dependencies
between them. Thus, a failure5 in one infrastructure may affect the operation of other CIs due
to their dependencies. In the case of geographical dependency, seemingly independent CIs may
be affected by a threat due to their physical proximity. The protection from such types of
dependency failures is an active and recent area of research, as shown by the numerous projects
on the topic, e.g. DIESIS (Usov, 2010; Rome, 2009), I2Sim (Marti, 2008), CIPRNet (Critical
Infrastructure Resilience, 2014). Dependency modeling, simulation and analysis (MS&A) has
been studied extensively, using various approaches. A recent publication (Ouyang, 2014)
overviews existing approaches and categorizes them in the following broad categories: (a)
empirical, (b) agent-based, (c) system dynamics based, (d) economic theory based, (e) network
based approaches, and others.
Disruptions or outages in CIs are usually categorized as cascading, escalating, or common-
cause (Rinaldi, 2001).
A cascading failure is defined as a failure in which a disruption in an infrastructure A
affects one or more components in another infrastructure, say B, which in turn leads to
the partial or total unavailability of B.
An escalating failure is defined as a failure in which disruption in one infrastructure
exacerbates an independent disruption of another infrastructure, usually in the form of
increasing the severity or the time needed for recovering from the second failure.
5 Failures in CI operation are meant in a broad sense including accidental failure, natural disasters
and deliberate cyber threats (Kozik, 2013).
Conclusions
73
A common-cause failure occurs when two or more infrastructure networks are disrupted
at the same time: components within each network fail because of some common cause.
This occurs when two infrastructures are co-located (geographic interdependency) or if
the root cause of the failure is widespread (e.g. a natural or a man-made disaster).
5.1.1. Large-scale and cross-sectoral dependencies
Known examples of large-scale failures caused due to CI dependencies are often related to
power transmission networks, such as the 2003 major blackouts in the US, Canada and Europe
(Andersson, 2005). The cascading process of a failure has been studied and modeled in the past
(Buldyrev, 2010; Panzieri, 2008; Vespignani, 2010; Zhou; Duenas, 2009; Zio; 2011). Although
there is a lack of available statistical data for such failures, recent efforts (Eeten, 2011), have
produced failure statistics based on empirical data reported to the media. One of the key
findings is that large-scale CI cascading dependencies occur more frequently than expected.
However, the effects do not often cascade deeply, i.e. nodes that are 4 or 5 hops away in a
dependency chain are rarely affected. Another key finding is that even though most reported
initiators of cascading effects are CIs belonging to the ICT and the Energy Sector, most of the
times the cascading effects are cross-sectoral, i.e. CIs of multiple sectors are affected. This
seems reasonable since the source infrastructures (ICT and Energy sector CIs) usually offer
vital services to many other CIs of different sectors, thus creating multiple direct (or 1st-order)
dependencies.
5.2. Building Blocks
This section briefly describes the two main building blocks used in the proposed
methodology: (a) the underlying multi-risk dependency analysis methodology for cascading
failures and (b) fuzzy modelling, which is used to for the time-based analysis of dependencies.
These blocks along with the proposed methodology were used to develop and test the tool
developed for the purposes of this dissertation, named CIDA (the Critical Infrastructure
Dependency Analysis tool)
5.2.1. Multi-risk dependency analysis methodology
In our previous work, a multi-risk dependency analysis method is presented (Kotzanikolaou,
2011; Kotzanikolaou, 2013; Kotzanikolaou, 2013) which makes use of the combined results of
existing organization-level risk assessments, already performed by the CI operators, in order to
assess the risk of n-order dependencies. To visualize the relationships (dependencies) between
CIs, we use directed graphs.
74
5.2.1.1. First-order dependency risk
Dependency can be defined as “the one-directional reliance of an asset, system, network, or
collection thereof –within or across sectors– on an input, interaction, or other requirement from
other sources in order to function properly” (NIPP, 2013). In our approach, dependencies are
visualized through graphs G = (N, E), where N is the set of nodes (infrastructures or
components) and E is the set of edges (or dependencies). The graph is directional marking
dependencies from one CI to another. An edge from node CIi → CIj denotes a risk relation,
deriving from the dependency of the infrastructure CIj from a service provided by infrastructure
CIi. To quantify this relation, we use estimations that quantify the resulting impact (Ii,j) and the
likelihood (Li,j) of a disruption being realized. The product of the last two values is defined as
the dependency risk Ri,j caused to infrastructure CIj due to its dependency on infrastructure CIi.
The numerical value in each edge refers to the level of the cascade-resulting risk for the receiver
due to the dependency. This risk is depicted using a risk scale [1..9], where 9 indicates the most
severe risk. All parameters (Li,j , Ii,j and Ri,j ) are defined in order to assess the risk of the 1st-
order dependencies. The main input of this method is provided by operators and refers to
obvious, upstream dependencies as mentioned above.
Example. The following case describes how our method can be used to model multiple 1st-
order dependencies. The cause of the mini telecommunication blackout in Rome in 2004
Kroger, 2010) was a flood in a major telecommunication service node in Rome due to a broken
pipe that provided water to the air conditioning plant. This caused several circuits to fail
including the main power supply. Alternative power generators failed to start due to the
presence of water and the batteries of electronic equipment soon followed. In order to perform
repairs, the air conditioning plant had to be shut down, which led to overheating of several
Telco node devices. This disruption in a main Italian Telecom infrastructure (node A) caused
problems and delays in different infrastructures (depicted in Table 11, based on the description
of (Kroger, 2010). These include the Fiumicino airport (node B) (closure of check-in, ticketing
and baggage services and transfers), the ANSI print agency (node C), post offices (node D),
banks (node E) as well as the ACEA power distribution (node F) and the communication
network (node G - both between landlines and between landlines and mobiles). In this example,
the airport operator (CIB) has a dependency risk RA,B from the infrastructure CIA. This risk
value refers to the likelihood of the disruption in the telecom node to cascade to the airport
(LA,B ), as well as the societal impact IA,B caused to the airport in the case of such a failure
been realized at the source of the 1-st order dependency, i.e. CIA.
Conclusions
75
5.2.1.2. Risk of n-order dependencies
Using the 1st-order dependencies as described above, we can proceed in assessing potential
n-order cascading risks based on a recursive algorithm (Kotzanikolaou, 2013). Let CI =
(CI1,...CIm) be the set of all the examined infrastructures. CIY0 → CIY1 →...→CIYn denotes a
chain of connected infrastructures of length n. The algorithm examines each CI as a potential
root of a cascading effect (denoted as CIY0 ) and then computes the Dependency Risk DR
exhibited by CIYn due to its n-order dependency.
Table 11. Example: Mini Telco blackout 1st-order dependencies
Let CIY0 → CIY1 → ... → CIYn be a chain of dependencies, LY0,...,Yn be the likelihood of the
n-order cascading effect and IYn−1,Yn be the impact of the CIYn−1 →CIYn dependency. The
cascading risk exhibited by CIYn due to the n-order dependency is computed as:
The cumulative Dependency Risk should consider the overall risk exhibited by all CIs within
the sub-chains of the n-order dependency. Let CIY0 →CIY1 → ...→CIYn be a chain of
dependencies of length n. The cumulative Dependency Risk, denoted as DRY0 ,Y1 ,...,Yn , is
defined as the overall risk produced by an n-order dependency, computed by the following
equation:
76
Equation 2 computes the overall dependency risk as a sum of dependency risks from each
affected node in the chain, due to a failure realized in the source node of the dependency chain.
Risk computation is based on a risk matrix that combines the likelihood and the incoming
impact values of each vertex in the chain.
Moreover, in several cases, the likelihood values are not easy to estimate or may not be
available. This means that while a dependency can be identified between two nodes, the
probability for a failure to propagate between two nodes is either unknown or certain (likelihood
= 1). In both cases, we can follow a simplified version of equation 2, shown in equation 3, that
follows the assumption that if a node fails, the dependent nodes will also fail (likelihood=1).
The n-order dependency risk is then calculated as the cumulative impacts on the affected nodes
in the dependency chain.
5.2.2 Fuzzy Logic
The multi-risk methodology described above is static in time, since equations 1 - 3 are based
on the maximum expected impact of each dependency. The values produced by these equations
assume that: (a) each dependency chain will always produce its worst case impact (and risk)
and (b) all dependencies will exhibit the same impact growth rate. However, in reality neither
all CI nodes of a chain will escalate to their maximum consequences nor will they experience
the same impact growth rate over the time. For this reason, we will extend our multi-risk
methodology to incorporate a dynamic, time-based analysis and to also assess scenarios of
partial failures. To model this behaviour, we will incorporate fuzzy set theory. We briefly
describe the basic concepts that we will integrate in our time-based analysis described in
Section 5.3.
In contrast to classical set theory, fuzzy logic is an attempt to find approximations of vague
groupings, in order to project more objective evaluations of values that are difficult to compute
(Perfilieva, 1999). Fuzzy logic variables provide a truth-value that ranges in [0, 1] for a possible
outcome. Basically, this truth-value corresponds to a membership percentage in a set. Our goal
is to approximate the time evolution of a cascading failure using fuzzy approximations of
impact evolution for various growth models, similarly to a real failure. For example, an incident
might at first have a slow cascading effect to other dependent CIs and as time goes by, failure
to restore its operation might lead to catastrophic effects.
Conclusions
77
5.3. A Method for Time-Based Analysis of Cascading and Common Cause
Failures
First, we extend the static dependency analysis methodology of (Kotzanikolaou, 2011) to a
dynamic time-based analysis model. We use different cascading failure growth models and we
apply fuzzy logic, in order to simulate realistic approximations of dynamic cascading failures.
In addition, we model combined cascading and common cause failures to simulate the effects
of dynamic, large-scale and major disasters.
5.3.1. Modeling Time Analysis of Cascading Failures
Recall from Section 2.1 that Ii,j, Li,j denote the impact (in a Likert scale) and likelihood (%)
of a failure experienced in the dependency CIi →CIj. These values are derived from assessments
performed at an organizational level by the CI operator. As in most static-based risk assessment
methodologies, the impact value Ii,j refers to the maximum expected impact (worst-case
scenario approach) regardless of the time it will take for maximum impact to fully realize after
a failure. To model the dynamic time-based analysis, we use the following steps:
1. Model Definition: Define different failure growth rates.
2. Setup: Using the growth rates, pre-compute all possible expected time-based impact
values.
3. Calculate fuzzy time-based impact values: For a given Dependency Risk Graph, use the
pre-computed expected time-based impact values as input to the fuzzy model, to output
the fuzzy approximation of the time-based impact for each dependency.
4. Assess time-related dependency risks: For each dependency chain, output the time-
based Cumulative Dependency Risk using the fuzzy time-based impact values.
These steps are described in detail in the subsections that follow.
5.3.1.1. Model Definition
Let Ti,j denote the time period that the dependency CIi →CIj will exhibit its maximum
expected impact Ii,j and let Gi,j denote the expected growth of the dependency failure, e.g. slow,
linear or fast evolution of the expected consequences, after the failure.
The values of Ti,j and Gi,j are provided by the risk assessors of each CI along with the values
Ii,j and Li,j . Finally, let t denote an examined time period after a failure. In the rest of this
Section and if there is no ambiguity, we will omit the dependency indices for simplicity reasons,
i.e. we will use I,T,G, instead of Ii,j, Ti,j, Gi,j.
The definition sets for all the above values are Likert scales defined as:
78
I ∈ [1..9], 1 is the lowest and 9 the highest impact.
T, t∈[1..10]. More specifically, we adopt a granular timescale, which uses the following
unavailability time periods: 1 = 15min, 2 = 1hour, 3 = 3h, 4=12h, 5=24h, 6=48h,
7=1week, 8=2w, 9=4w and 10=more (> 4w).
G ∈ [1..3], where 1 represents a slow, 2 a linear and 3 a fast growth (evolution) of a
failure experienced in the examined dependency.
Definition 1. Let C Ii → C Ij be the examined dependency with maximum expected impact I,
experienced at time period T after a failure and let G be the growth evolution of the failure.
The expected time related impact of the dependency, experienced at time t is defined as:
Obviously, I(t) = I for t ≥ T if no mitigation controls are taken at the nodes. Using equation
4, each dependency chain can be modeled with the most appropriate evolution growth and a
different model is used for each value G. Fast cascading effects are considered to escalate
logarithmically (fast short term growth that stabilizes to the maximum expected impact). Linear
cascading effects are considered to escalate following a typical linear equation. Slow cascading
effects follow an exponential growth rate, which starts with low initial values and escalates on
the last steps of the time scale. Figure 16 presents a comparison of the different growth rates,
for the same T and I.
Figure 16. Example of the expected impact evolution for different growth rates, for I = 9 and T =
4weeks
Conclusions
79
Example: In a Dependency Risk Graph, the dependencies of CIs connected with a nuclear
power facility are modeled with very high impact (I = 9) experienced at a relative short time
period (T = 3h) with a fast evolution (G = 3). In the same graph, the edges starting from another
typical energy provider are modeled with impact 5 experienced at 48h after the failure and with
a linear evolution (G = 2). Our model will be able to project the evolution of the dependency
risk paths for all time periods, using all different models.
5.3.1.2. Setup
The growth rates modeled by equation 4 are used to pre-compute all possible values for I(t),
for all possible combinations of I,T,G (see Algorithm 1).
The output of this algorithm will be used as input to a fuzzy logic ranking system, to assess
more realistic evolutions of potential failures. The fuzzy logic classification system uses the
following membership sets:
1. The impact set: partitions the [1..9] impact scale to groups Very Low, Low, Medium,
High and Very High as: {VL={1}, L={2,3}, M={4,5}, H={6,7}, VH={8,9}}.
2. The time set: partitions the [1..10] time scale to groups Early, Medium, Late and Very
Late periods as: {E={1,2,3}, M={4,5,6}, La={7,8}, VLa={9,10}}.
In order to support the fuzzy mechanism (described in Section 3.1.3), the output of Algorithm
2 (as depicted below) is stored in pre-computed tables6. These provide the expected time related
impact values for all possible G and T.
6 All tables are available at the tool’s Wiki: https://github.com/geostergiop/CIDA/wiki/ and in
Appendix A.
80
Algorithm 1. Calculation of all possible I(t) values
An example is shown in Table 12 below, which describes all possible time-based impact
values Ii,j (t), assuming a fast evolving failure (G = 3), experiencing the worst-case impact at
time T = 12h, depicted by the yellow horizontal line. Each column’s worst-case impact value
is appointed to cells in that row (T = 12h) and obviously to all rows below that one, since in
this scenario the worst-case impact has already been realized at that time. The time-based
impact values for all the rows above this (T =15min up to T = 3h) are calculated by applying
an inverse growth rate on the cell values in the T = 12h row, using equation 4 for G=3. Notice
that the impact and time values have been grouped according to the fuzzy impact and time
membership sets respectively. For example, in Table 6, the fuzzy impact membership set ‘Low’
Conclusions
81
contains impact values from 1 up to 3. All the values have been produced by applying the fast
(logarithmic) growth scale and the time T = 12h as the expected time of occupancy of the worst-
case impact value. The output of the algorithm is computed during the setup and it is stored in
30 tables: for each of the three growth rates G, we need to pre-compute and store one table for
each of the ten possible T values.
Table 12. An example of a pre-computed table describing the expected time related impact values for
fast evolving failures (G = 3) exhibiting their worst impact at T = 12 hours
5.3.1.3. Calculating Fuzzy Time-based Impact Values
By using the pre-computed tables with all expected time-based impact values I(t), it is now
possible to assess the fuzzy estimation of the time-related impact values, for a given
Dependency Risk Graph.
For each dependency, we use the growth rate G and the expected time T of the worst-case
impact value I, to select the corresponding table from the database. Then, fuzzy sets for all
impact membership sets (Very Low, Low, Medium, High, Very High) are generated, using the
corresponding columns of the selected table. These fuzzy sets along with linguistic IF-THEN
rules are used to calculate the fuzzy value of the expected time-based impact value. Rules are
usually expressed in the form: ”IF variable IS property THEN action”. All IF-THEN rules are
invoked; using the constructed membership sets as linguistic variables to determine an output
result, the fuzzy time-based impact value, denoted as:
𝐹𝑢𝑧𝑧𝑦( 𝐼, 𝐺, 𝑇, 𝑡) = ℐ(𝑡) (5)
The calculation of ℐ(𝑡) is computed as follows. Initially a processing stage invokes the
appropriate IF-THEN rules 7 and generates a result for each rule. Then these results are
combined to output a set of truth values. Each IF-THEN result is, essentially, a membership
function and truth value controlling the output set, i.e. the linguistic variables impact and time.
The final stage in order to get a single quantitative value from the fuzzy output of the
7 See Appendix A for a more thorough presentation of the IF-THEN rules used.
82
membership values is “defuzzification”, in which all IF-THEN output results are combined to
give a single fuzzy time-based impact value for each time point in the time scale. We use the
RightMostMembership defuzzification technique (Leekwijck, 1999), which outputs the most-
right (i.e. highest) impact values, since this is coherent with risk-based standards that tend to
favor worst-case scenarios.
The output fuzzy time-base impact values ℐ(𝑡) are considered more objective
approximations of the expected impact at a given time, since instead of simply using the
appropriate pre-computed expected time-based impact I(t), the fuzzy values also consider their
neighboring values. Thus they tend to be closer to real-world scenarios. In short, each
dependency has its own expected growth but it will also be affected by the growth of its near
dependencies.
Example: An examined dependency has input data G=3 and T =12h, thus the fuzzy
mechanism will select Table 2. The group Low of the fuzzy impact set (columns 3 and 4)
contains only values 1, 2, 3. By using the aforementioned mechanism, the fuzzy membership
set Low has membership percentages defined as: Low = {(1, 0.05) (2, 0.55) (3, 0.4)} where the
second value is each couple is the membership percentage of the corresponding impact value.
The subset of the rules used to calculate the Low output set of the values ℐ(𝑡) are the following:
17: IF Impact IS Low AND Time IS Early THEN Fuzzy Impact is Very Low;
18: IF Impact IS Low AND Time IS Medium THEN Fuzzy Impact is Medium;
19: IF Impact IS Low AND Time IS Late THEN Fuzzy Impact is High;
20: IF Impact IS Low AND Time IS Very Late THEN Fuzzy Impact is High;
Let us calculate the fuzzy impact for I = 2. The fuzzy set mostly characterized by that value
is the Low set. Let us also assume that the worst-case scenario for this example is at T = 12h.
Based on Table 2, this time belongs to the Medium time fuzzy set. Thus, using rule 18, CIDA’s
setup process will choose to output a Medium fuzzy time-based impact value. Last but not
least, the tool will now use the aforementioned RightMostMembership defuzzification
technique on all these sets to discrete the time-based impact value.
5.3.1.4. Time-related Multi-order Dependency Risk
The static model described in Section 5.2.1 can be now extended to provide multiple
estimations of the evolutions of the dependency risk, by replacing the static impact value Ii,j,
with the dynamic fuzzy time-related impact values ℐ(𝑡) described above. These values are used
to extend equations 2 and 3 to calculate the n-order dependency RY0 ,...,Yn and cumulative Risk
DRY0 ,...,Yn of a risk graph for each point on the time-scale as:
Conclusions
83
𝐷𝑅𝑌0,…,𝑌𝑛(𝑡) = ∑ 𝑅𝑌0,…,𝑌𝑛
= ∑ (∏ 𝐿𝑌𝑗−1,𝑌𝑗
𝑖𝑗=1 ) ∗ ℐ𝑌𝑖−1,𝑌𝑖
(𝑡)𝑛𝑖=1
𝑛𝑖=1 (6)
or, if likelihood assessments are omitted:
𝐷𝑅𝑌0,…,𝑌𝑛(𝑡) = ∑ 𝑅𝑌0,…,𝑌𝑛
= ∑ ℐ𝑌𝑖−1,𝑌𝑖(𝑡)𝑛
𝑖=1𝑛𝑖=1 (7)
Obviously, equations 6 and 7 will produce ten different values, one for each different
examined value of t. Examples of test scenarios are given in Section 5.6.
5.3.2. Combining cascading and common-cause failure risks
The Dependency Risk assessed by equations (6) or (7), assumes a single initiating event
(disruption) at a single CI that results in cascading disruptions. It does not cover common-cause
failures which simultaneously affect several, seemingly independent CIs. Such events can cause
multiple cascading chains, i.e. impact is introduced to multiple nodes in the graph
simultaneously. Thus, we extend the model to also capture failures that are at the same time
common-cause and cascading failures. A variety of incidents can serve as initiating events of
such cases: An accident or a natural disaster, or it can be a human-initiated attack. For example,
a common-cause initiating event may concurrently affect CIs –not been identified as directly
dependent to each other– due to physical proximity (e.g. a flood) or due to social parameters
(e.g. a national strike).
Let Le be the likelihood of an event (threat) e. In the case of natural disasters, the value of Le
can be assessed based on statistics of previous incidents, prognostics and the presence of
vulnerabilities, whereas the likelihood of adversarial attacks is more complex. In that case, the
motivation and the attack skills of the adversary, as well as his perceived impact of the attack,
affect likelihood. For this reason, the use of expert opinion is commonly applied, coupled with
a worst-case approach, to achieve a maximum valuation of risk.
We first use equation 6 (or its simplified impact-only version of equation 7) to evaluate all
possible n-order dependency risks. Let ℂ𝕀 be the set of all the examined CIs. The combined
Common-cause Risk, CR(CIY0 , e) of all possible chains of cascading events C IY0 → C IY1 →
C IYn , initiated by a common-cause failure event e, for each possible source infrastructure CIY0
∈ ℂ𝕀 can be computed as the sum of all the possible risk chains DRY0 ,...,Yn , ∀ Y0 ∈ ℂ𝕀,
multiplied by the likelihood Le of each examined event e:
𝐶𝑅(ℂ𝕀, 𝑒) = 𝐿𝑒 ∗ ∑ 𝐷𝑅𝑌0,…,𝑌𝑛(𝑡)∀ 𝑌0 ∈ ℂ𝕀 (8)
Every CI (node) that is affected by a common-cause event e is examined as a possible root
of a dependency chain (as CIY0 ). For each CIY0 , the cumulative dependency risk DR is
computed by applying equation (6) or the simplified version (7). Examples of the combined
Common-cause Risk evaluation are provided in section 5.6.3.
84
5.4. Design and Implementation of the CIDA tool
In this section we describe the building blocks used for the design and implementation of the
Critical Infrastructure Dependency Analysis (CIDA) tool (Stergiopoulos, 2014), which extends
and implements the dependency analysis methodology described in section 5.3.
5.4.1. The Neo4J graph database
We chose a graph database model as the main building block for the development of CIDA.
Graph databases are defined as storage systems that provide index-free adjacency. Graph
database technology is an effective tool for modeling data, in comparison with relational
databases and querying languages, in cases where the relationship between elements is the
driving force for the design of the data model (Vicknair, 2010; Shao, 2012). In a graph database,
every element (or node) only needs to know the nodes with which it is connected (i.e. its edges).
This allows graph database systems to utilize graph theory in order to efficiently examine the
connections and degree of nodes’ connectivity. In addition, the edge utility allows a graph
database to find results in associative data sets. Graph databases can scale more naturally to
large data sets or to datasets with frequently changing or on-the-fly schemas (Vicknair, 2010).
We examined various graph databases and we selected the Neo4J framework to implement
our dependency analysis-modeling tool, due to its adaptability, scalability and efficiency.
According to recent empirical studies (Shao, 2012; Jouili, 2013; Batra, 2012)), Neo4J
outperforms other systems in load time for thousands of elements, as well as in the time needed
to compute the total paths and detect the shortest path. On the other hand, Neo4J shows inferior
performance when used in highly volatile network topologies (i.e. graphs with frequent changes
in nodes and edges), in comparison to other graph model approaches (e.g. DEX, Titan-
Cassandra). This however does not affect our model, since CI dependencies are not subject to
frequent changes in their connectivity.
Neo4J builds upon the property graph model; nodes may have various labels and each label
can serve as an informational entity. The nodes are connected via directed, typed relationships.
Both nodes and relationships hold arbitrary properties (key-value pairs). Although there is no
rigid schema, the node-labels and the relationship-types can provide to the nodes as much meta-
information as necessary for the node attributes required by a specific schema. When importing
data into a graph database, the relationships are treated with as much value as the database
records themselves (Vicknair, 2010).
Neo4j deploys a single server instance that can handle a graph of billions of nodes and
relationships. If data throughput exceeds a limit, the graph database can be distributed among
Conclusions
85
multiple servers thus providing a scalable configuration with high availability. In addition,
Neo4J’s listeners are capable to capture useful signals (notices) that objects broadcast in
response to possible events, such as a change of a property value or a user interaction. As
explained in Section 5, these two graph functions are the most important factors affecting the
computation time for the analysis of dependent and interconnected CIs.
During development, the Blueprints technology was also used: a property graph model
interface, which provides implementations to support the Neo4J graph database.
5.4.2. Design of the Dependency Graph Analysis Tool
We developed CIDA, a graph-based Critical Infrastructures Dependency Analysis tool that
implements the methodology described in section 5.3, in order to dynamically analyze the risk
of CI dependency chains, for cascading and/or common-cause failures. CIDA is able to
compute the security risk and/or the impact evolution of the dependencies over the time. CIDA
can graphically represent complex graphs of thousands of dependent CIs through a weighted,
directed graph. The weight of each connection (edge) between two CIs, is the (maximum)
estimated dependency risk value deriving from the dependency between two infrastructures. In
order to make the computation of the risk dependency paths more efficient, we need to set a
maximum depth limit of the examined dependencies. Based on recent empirical research results
(Eeten, 2011), cascading effects rarely affect CIs beyond 5th-order dependencies and thus we
use this as an upper limit to the order of dependencies that are evaluated.
The tool takes as input (either from a spreadsheet or using a graphical interface) the nodes
and the edges of the graph. For each edge CIi →CIj the following input is required for a static
analysis: the estimated likelihood Lij and the (maximum) expected impact Iij. For a dynamic
time-based analysis, the expected time Tij of the maximum impact and the expected growth rate
Gij for the dependency are also required. For a given input dependency graph, CIDA outputs
the following:
1. A table of all existing dependency paths, up to a given maximum dependency order (5-
th order by default).
2. For a static analysis (based only on Lij , Iij ) and for each dependency path, it computes
the Cumulative Dependency Risk using equation 2.
3. For a dynamic analysis (additionally using the time related input Tij , Gij ) it computes
the expected Cumulative Dependency Risk for various time frames, using the formula
of equation 6.
4. The dependency paths can be sorted based on their cumulative Dependency Risk value
and can also be presented in a graphical form. If the assessor has set a maximum risk
threshold, then CIDA also highlights all the paths exhibiting a value higher than the risk
86
threshold. The paths can be exported to several forms, such as a spreadsheet or an XML
file for further analysis.
5. In addition, CIDA graphically highlights the path exhibiting the maximum cumulative
Dependency Risk path, using a modified version of the Dijkstra algorithm. The
algorithm uses negative weights (risk values) to compute the maximum weighted path.
Based on the assessor’s preferences, CIDA can compute either cyclic paths (if feedback
effects are to be considered) or acyclic paths (if feedback effects are excluded). In cases where
the likelihood values are not available, CIDA can proceed using only the impact values and
equations 3 and 7 can be used for a static or dynamic analysis of dependency impact paths
respectively. Obviously, impact estimations will be higher than risk estimations, as they
represent worst- case scenarios that take for granted that if a node fails, then all the following
nodes will experience a total failure.
5.4.3. Modeling CIs & Dependencies
In CIDA, each node may represent a CI or an “autonomous” sub-component of a CI (e.g. a
power generation sub-station in the case of a power operator), depending on the required level
of analysis. Each node in a graph supports the following attributes:
Name: A unique name for the node.
CI operator: The CI owner, responsible for the proper operation of this node. If the
analysis is performed in a unit-level, then one operator may be responsible for several
nodes.
CI sector: The sector of the CI operator. For example Energy or ICT sector.
CI sub-sector: The specific sub-sector that the operator of the CI belongs to. For example
Electricity production or Telecom operator.
Node Location: (Location latitude and longitude) Used to capture geographical
dependencies between CIs and evaluate potential threats concurrently affecting several
nodes.
maxPath: A Boolean that simple indicates if the node belongs to the maximum risk path.
CIDA supports 17 different CI sectors, including communications, energy, transportation,
water systems and all the recognized CI sectors, based on the reports of (Presidential Policy
Directive, 2013) and (Commission of the European Communities, 2005). It also allows
modeling all types of dependencies. While logical, informational and physical dependencies
may be defined in Service Level Agreements (SLAs) and, thus, are easier to identify,
geographical dependencies may be missed. CIDA can automatically identify geographical
Conclusions
87
dependencies, based on the location provided for each node. This enables CIDA to study threat
scenarios within a specific geographical region and thus to assess geographical dependencies.
Based on the appropriate formulas described in section 5.3, CIDA computes the risk for each
individual dependency path. Then the risk values are then stored in the Neo4J graph, which is
implemented as a weighted, directed graph.
Figure 17. Calculating a dependency risk graph for a test scenario with 100 nodes
It then creates a visualization interface using the JUNG2 graph visualization library,
supported by Blueprints and Neo4J.
An example of a graph output is shown in figure 17. Each node represents a CI and the weight
of each edge is the estimated dependency risk value that derives from the dependency between
two nodes. The type of the dependency is also depicted. Red edges and nodes indicate the
maximum cascading risk path. The complete set of paths and the relative risk values are
exported to a spreadsheet, for further analysis. Examples of complete input and output sets are
provided in Section 5.6.
5.5. Efficiency analysis
We empirically study the efficiency of the CIDA dependency analysis tool presented in
section 5.4 using granular random scenarios as test cases, containing from 10 up to 1000 nodes.
All the tests were performed on an Intel Core i-7 processor @2.7GHz with 4 cores and 16 GB
of RAM. For each scenario, we examined two different cases with different degrees of
connectivity. In the low connectivity case, each node is randomly connected with at least 1 and
at most 3, other nodes. In the high connectivity case, each node is randomly connected with at
least 4 and at most 5 other nodes. For each case, we multiplied the execution time by a factor
88
of 10, to estimate the execution time for dynamic analysis, since for time-based analysis all the
computations are repeated for each examined time frame.
In real-world scenarios the majority of nodes are expected to have a connectivity degree ≤ 3,
while only a small portion of nodes is expected to experience a high degree of connectivity (e.g.
major electrical production sub-stations or key Telco node) (Eeten, 2011). Thus it is reasonable
to expect that in a real-world scenario the average connectivity degree of all the nodes will be
between these values and the expected execution time will be between these setups (for the
same number of nodes).
Each test was repeated 10 times and the mean execution time was considered. The execution
time includes the calculation of all the dependency risk paths up to 5th-order dependencies, and
the time required to short the paths according to the execution time and compute the maximum
dependency risk path. Again, since the cascading effects rarely affect nodes at a distance greater
that 4 hops away from the source of the event, the computation of up to the 5th-order
dependencies suffices to cover the majority of the cases (Eeten, 2011). As expected, the
computation time of the complete set of existing dependency risk paths increases exponentially
with the number of nodes (see figure 18). Also, the connectivity degree seems to significantly
affect the execution time. For example, in the scenario of 1000 nodes it takes about 20 min in
the case of low connected nodes (1-3 edges per node), while it requires more than 332 min for
higher connectivity (4 − 5 edges per node).
Since the maximum path problem is NP-complete, it is obvious that CIDA cannot provide a
complete theoretical solution. However, with reasonable limitations in system parameters such
as the maximum number of nodes, the degree of node connectivity and the degree of the
dependency order, CIDA can efficiently provide useful results for large-scale and cross-
sectorial realistic scenarios.
Conclusions
89
Figure 18. Execution time vs number of nodes
5.6. Analyzing Real-World Scenarios of Cascading Effects
To demonstrate the applicability of CIDA, we run scenarios based on a known real-world
case of cascading effects. Although our tests are based on a real case, the impact, likelihood
and time-related input assigned to each dependency do not rely on actual risk assessment
results, since such assessments are not publicly available, but they are based on our subjective
assumptions for demonstration purposes.
5.6.1. Case study: A real-world cascading blackout failure
We run a scenario consisting of 9 nodes, based on the well-known electricity blackout
scenario of California (Rinaldi ,2001). The scenario was selected because it is a well-
documented case, which exhibits several complex cross-sectorial and multi-order cascading
dependencies. The initiating event in this scenario is the failure of an electric power sub-station
(node A –see figure 19).
90
Figure 19. Dependencies of the California blackout scenario
This event triggered the following 1st-order dependencies: the disruption to a natural gas
production infrastructure (node B), the disruption to the operation of petroleum pipelines (node
E) transporting jet fuel within neighbor states, and problems in the operation of massive water
pump units (node H).
As depicted in figure 19, the disruption to gas production (node B) directly impacted gas
supplies for steam injection units (node C, – 2nd-order dependency). The steam injection units
affected the operation of heavy oil recovery units (node D, – 3rd-order dependency), further
exacerbating power problems to node A (feedback loop). Similarly, the disruption of product
petroleum pipelines (node E) caused inventories to build up at refineries and draw down at the
product terminals (2nd-order dependency), including several major California airports (node
F). The reduction of jet fuel stocks at the airports caused several major airline operators (node
G) to consider contingency plans (3rd-order dependencies). Finally, the disruption of the water
pump units (node H) affected crop irrigation at crop fields (node I, –3rd-order dependency).
For our case study, we used as input the values shown in table 3. For each dependency, we
provide a Likelihood and a maximum expected Impact estimation. We also provide the
expected time that the maximum impact will manifest, as well as estimation for the growth rate
of the failure evolution.
Note that the input data of the 1st-order dependencies can be fed to CIDA either through a
spreadsheet, as the one shown in table 13, or they can be graphically added/modified (an
example is shown in figure 20). Based on the data input, CIDA will compute the complete set
of all the dependency risk paths in a time-axis, for each dependency chain of order ≤ 5, using
the formula of equation 6.
Conclusions
91
Table 13. Assumed input values for the examined blackout scenario.
CIi CIj Li,j Ii,j Ti,j Gi,j
A B 0.9 8 24h fast B C 0.65 6 48h linear C D 0.7 6 48h slow D A 0.15 3 2w slow A H 0.8 6 12h fast H I 0.65 7 48h linear A E 0.9 8 12h fast E F 0.7 5 24h slow F G 0.25 8 1w slow
Figure 20. Using the graphical interface to input data for nodes (CIs) and connections (edges)
CIDA will output a graphical representation of the examined Dependency Risk Graph, as
shown in figure 21. In this case, the graph of the 9 examined nodes produces 38 chains of order
between 2 and 5 and of potential risk value between 1.5 and 13.17. The nodes and edges
indicated with red color show the maximum Cumulative Dependency Risk path.
92
Figure 21. The Dependency Risk Graph of the examined blackout scenario
5.6.2. A cascading-only dependency failure scenario
After CIDA has evaluated all the Dependency Risk paths, it is now possible for the assessor
to efficiently examine several scenarios. One scenario is to analyze and compare all possible
cascading effects. CIDA produces a list of all dependency paths, sorted by their Cumulative
Dependency Risk value. This may help the assessor to identify all the potential dependency
risks that are above a threshold value. For example, figure 22 below shows the subset of the
dependency risk paths that exhibit a cumulative risk above 5, regardless of the time required to
reach this threshold. It also shows the risk values of the maximum risk path for all the examined
time periods, as well as the maximum risk level for the rest paths at the time of occurrence.
Conclusions
93
Figure 22. Dependency risk paths with Cumulative Dependency Risk above a risk threshold equal to
5. The assessor may project different paths by using a different threshold
The threshold parameter is chosen by the assessor and may assist the decision makers in
implementing the most effective risk mitigation strategies. From the above figure it is easy to
see that the four dependency paths with the highest risk value, surpass the threshold only within
1 hour after an initial failure and they all start at node A. Thus a cost-effective mitigation
strategy would start by applying mitigation controls at node A with fast response time, which
would substantially decrease the overall dependency risk. If mitigation techniques at node A
are considered too expensive, an alternative strategy would be to reduce the likelihood of
cascade to the most important 2nd-order dependencies of node A (nodes B and E).
A second result of the analysis is that, although path (A-B-C-D) exhibits the highest risk for
almost all examined time frames, still we can see from the graph that path (A-E-F-G) is the
most critical path around 12 hours after a cascading failure. This is due to the fact that although
both dependencies A-B and A-E have a fast growth, the second dependency is expected to have
the fastest convergence to its maximum impact (TAE =12h). Recall that our methodology can
model different Tij and Gij for each dependency.
A third result can be derived by comparing the evolution of sub-paths exhibiting high risk.
For example, note that although the path (A-B-C-D) is the highest risk path, its sub-path (A-B-
C) already exhibits impact higher than the threshold within 1 hour. Thus it is necessary to
implement mitigation controls at the first or second order dependency.
Finally, for the rest of the dependency paths (A-H-I, B-C-D-A-E-F and C-D- A-E-F-G) it is
safe to consider mitigation controls with a slower response time, since they have relatively low
risk, even for long time frames. Note that the last two paths include the chain (A-E-F) as sub-
94
path thus mitigation controls implemented at (A-E-F) will concurrently reduce the risk of 4 out
of the 7 most critical paths.
5.6.3. Combined common-cause and cascading scenarios
Another scenario is to analyze all the cascading effects that may potentially be triggered by
a common-cause failure. For each examined node, CIDA computes the sum of the dependency
risks of all the existing distinct paths8 originating from it. In the examined case, node A is (as
expected) by far the most critical node for common-cause failures, having a sum of distinct risk
paths equal to 33.22 (paths A-B-C-D, A-E-F-G and A-H-I). In a common cause scenario, at
least two nodes are directly affected by the initiating event, which serves as the common cause
for the failures. Therefore, for each affected node, we would calculate the sum of distinct risk
paths and then these values would be weighted with the likelihood value Le for the initiating
event e (failure or attack) that may be realized at these source nodes. The combined risk (for
each possible initiating event e at each directly affected node) is computed using the formula
of equation 8.
Note that the complete set of the dependency chain risks has already been computed as an
output of the CIDA tool. Thus the evaluation of the possible common-cause failures will be
based on “ready to use” risk chains. The assessor is able to add initiating events and likelihood
values for every node, reflecting the probability that the examined event will cause a failure to
the node or not. The examined initiating events e and likelihood values Le are selected by the
assessor based on expert opinions and possible statistical data. Obviously it is reasonable to
first examine nodes that exhibit the higher sum of risk values (before they are weighted with
Le).
5.7. Discussion and Comparison with Other Approaches
5.7.1. Comparison with related methodologies and tools
Modeling and simulation approaches are distinguished as (a) empirical, (b) agent based, (c)
system dynamics based, (d) economic theory based, (e) network (topology or flow) and (f)
others (hierarchical holographic modeling, high level architecture based method, petri-net
based method, dynamic control system theory based, Bayesian network based, etc.) (Ouyang,
8 In order to avoid repetition of the same risk chain, only distinct paths are considered.
Conclusions
95
2014) 9. Our method can be categorized as a hybrid one, having characteristics of empirical
methods (as a risk based approach) and network based methods (as a graph modeling tool).
Empirical methods have been criticized for the lack of the required statistical data needed to
assess the likelihood of potential events. While probability data may be difficult to collect for
various CIs, efforts have already been made for specific sectors. For example, studies of
blackouts allow the identification of critical power lines (or groups of power lines) for a given
network model (Carreras, 2012). This provides a technique for identifying critical clusters of
lines that are likely to trigger or propagate cascade effects, due to vulnerabilities of their power
lines.
Our approach also inherits from network-based methods as it combines a method for
discovering dependency risk paths with an automated modeling and analysis tool. It allows the
dependencies of the connected infrastructures to be depicted in a graph and critical paths to be
identified. Such flow-based, network approaches exist in the literature. They either model the
flow of products or services between CIs in a uniform model (Lee, 2007; Svendsen, 2007;
Svendsen, 2007) or they combine different sector-based flow models (Ouyang, 2011; Rosato,
2008).
Regarding the level of analysis, CIDA does not model infrastructures on a component level,
and for this reason it has similarities with the empirical Leontief input-output models used for
high-level multi-sectorial risk assessment (Haimes, 2001; Santos, 2004; Santos, 2006; Setola,
2009). These approaches measure the dependencies among CI sectors by economic
relationships. The approach described in (Setola, 2009) considers domain expert opinion and
uses fuzzy logic to assess the impact induced by direct and higher-order dependencies between
CIs. These models assume that the CI operators (or the expert(s) conducting the assessment)
will provide input data regarding the impact values for resource outages of various durations
on each CI. Both CIDA and (Setola, 2009) use fuzzy logic. The approach of (Setola, 2009) uses
it to minimize the uncertainty and ambiguity associated with the subjective information
obtained from domain experts. On the other hand, CIDA combines fuzzy logic with various
time growth models. Each dependency may follow a different growth rate and fuzzy logic is
used in order to objectify the evolution of each dependency, taking into consideration the state
of other near dependencies. This allows CIDA to output results both for various time frames
(this is also available in (Setola, 2009)) and for alternative growth rates of the failure. Moreover,
CIDA is based on a dependency risk graph to model dependencies, which are not limited to
economic dependencies.
9 Another taxonomy of dependency models contains six broad categories (Zio, 2011; Kroger,
2011): (a) aggregate supply and demand tools, (b) dynamic simulations, (c) agent-based models, (d)
physics-based models, (e) population mobility models and (f) Leontief input-output models
96
The approaches of (Haimes, 2001; Santos, 2004; Santos, 2006) use the input-output
inoperability model to assess the dependencies between the various sectors of an economy and
forecasting the effect on one segment of a change in another one, due to a disruptive event. Our
approach is not a purely economical one. Another important variation is that CIDA allows
alterative graphs to be created to analyze the dependencies that occur in abnormal operating
conditions; in contrast, the input to the approaches of (Haimes, 2001; Santos, 2004; Santos,
2006) only measure dependency in normal economic operations. Moreover, CIDA can perform
time-based analysis, which offers different risk results according to the time frame studied and
the rate that the impact evolves in each node. Regarding the input data required for the analysis,
the above approaches use aggregated economic data on a sector basis, while CIDA’s input
consist of risk assessment data provided by the CI operator, i.e. per each infrastructure.
Another economic based approach is the NISAC tool N-ABLE, which is a large-scale
microeconomic simulation tool that models the complex supply-chain, spatial market
dynamics, and CI interdependencies of businesses in the U.S. economy. N-ABLE has been
designed in particular to model how U.S. businesses can adapt to and recover from disruptive
events (Ehlen, 2005). Our tool is not specifically engineered to model the economic impact on
a microeconomic level.
The CIP/DSS (Critical Infrastructure Protection/Decision Support System) (Bush, 2005;
Conrad, 2006) enables decision makers to determine what consequences might be expected
from disruptions to infrastructure, explore the mechanisms behind these consequences, and
evaluate mitigation controls for a particular risk. CIP/DSS assesses uncertainties in threats,
vulnerabilities, and the consequences of terrorist acts and natural disasters at a national and
metropolitan scale. It models interdependencies for all U.S. identified critical infrastructures
and key resources and calculates the impact that cascade into these interdependent
infrastructures and into the national economy. Our tool could benefit by the CIP/DSS
representation of mitigation alternatives, in order to formalize more the selection of mitigation
strategies.
Based on our analysis, CIDA can efficiently compute the risk of all the dependency risk
paths, using reasonable limits for the length of the dependency order. However when examining
large-scale scenarios of hundreds of nodes, the execution time may not be feasible for real-time
analysis and response. The tool CIPR/Sim is such a real-time analysis tool, which imports real-
time data from numerous existing analysis modules, including RTDS (Real Time Digital
Simulator) for electric grid analysis, QualNet for telecommunications analysis, and PC Tide for
wind speed and flood surge analysis.
Conclusions
97
5.7.2. Advantages and Limitations
CIDA is a modeling and analysis tool focusing on the study of large-scale dependency
scenarios for proactive analysis. The primary goal of CIDA is to help risk assessors and CIP
decision makers to assess dependency risks proactively, before an initiating threat has been
realized. By analyzing the complete set of the potential dependency paths, the risk assessors
may project all the cascading effects that may potentially be realized and thus flag dependency
risks above a threshold that need further attention.
In addition, CIDA can be used to run specific scenarios, which may be of particular interest
for the risk assessors. While the exhaustive computation of the complete set of dependency risk
paths may provide useful information and reveal “hidden” dependency risks, the assessors may
also use CIDA to specifically examine particular realistic scenarios. For example the risk
assessors may use CIDA to examine “what-if” scenarios that only consider initiating security
events affecting one (or some) nodes. Such a scenario may include the study of a major physical
disaster, initially affecting all the nodes within a geographical range. This can be easily
implemented, since the attributes of each node in CIDA may also include the geographical
location coordinates. Thus it is possible to assess scenarios of common-cause failures to
targeted nodes and examine cascading effects based on geographical dependencies (in such a
case, the most usual type of the corresponding 1st-order dependencies will be geographical).
CIDA may also be used as an efficient tool for the proactive assessment of risk mitigation
controls and therefore, increasing resilience. Based on real input data, it is feasible to examine
hundreds of variant scenarios, even past incidents. The risk assessors may efficiently run slight
variations of dependency graphs, with different weights or even different dependencies, in order
to simulate the implementation of alternative risk mitigation controls. For example, if a
particular path has been identified by the exhaustive computation as a path of a dependency
risk above the maximum risk threshold, CIDA can be used to project the effect of implementing
redundancy security controls to decrease the dependency impact to a targeted edge; or the effect
of completely substitute an edge (if security controls that completely remove a dependency
were implemented), in order to optimize the topology of the interdependent CIs. Both examples
increase the absorbing capacity of nodes or of the network of CIs and, thus, increase overall
resilience.
CIDA can also be used in order to identify and target key nodes, in order to make them more
resistant to failures or improve their restorative capabilities, and which are the alternative
resilience parameters that could be improved. In this way it will be possible to evaluate the
benefit of various alternative and/or complementary mitigation controls, and output convincing
arguments concerning the expected benefit of possible mitigation strategies. Note that using
additional computing resources can parallelize the study of such scenarios.
98
A limitation of our approach is the need for input data of prior risk assessments, performed
by the examined CIs. This is an inherent problem of all the empirical risk approaches, since
empirical risk-based approaches analyze dependencies based on previous incidents (historical
incident or disaster data) coupled with expert opinion in an effort to identify alternative
measures to minimize the dependency risk (Utne, 2011; Kjolle, 2012). It is highly unlikely for
a single CI operator to have access to real data of other CIs. Thus the methodology can only be
applied at a higher layer. For example sector coordinators or regulators may collect data
concerning a specific sector (such as ICT or energy). National CIP authorities or CERTs may
be able to collect such information. Also note that input data can be gradually added in the
CIDA database in order to gradually construct large-scale scenarios. Moreover, it is also
possible to use such a tool based only on the expected impact of dependent CIs, and ignoring
the likelihood parameter, which is generally more difficult to collect (Franchina, 2011). We
plan to enrich CIDA 6in its next version with available statistical data for potential initiating
events and their likelihood of occurrence, for key sectors such as Energy and ICT, in order to
further assist risk assessors in evaluating combined common-case and cascading failures.
The entire CIDA project (both the stress test implementation along with the full GUI CIDA
tool can be found at: https://github.com/geostergiop/CIDA.
Conclusions
99
Chapter 6: Conclusions
6.1 Summary and discussion
This dissertation provides evidence suggesting that more work is required for the protection
of Critical Infrastructures, both in aspects of software security through user interaction and that
of external threats affecting CIs due to their dependencies.
Firstly, on a smaller granularity level, research findings suggest not only that automatic
analysis of source code for detecting logical errors is indeed feasible (contrary to popular
belief), but also that it can be used to detect software threats inside information systems and
predict possible infrastructure failures due to their manifestation. This way, a partial risk
estimation of the IT systems concerning an infrastructure can be provided. This research’s
potential impact is even, since academic literature has revealed that smartphone apps violate
the principle of least privilege and also suffer from privilege escalation vulnerabilities.
As discussed in §4, users’ actions can lead to vulnerabilities that, contrary to a priori known
faults, stem from a discrete sequence of valid actions. Moreover, experience has shown that
these types of faults cannot be detected through normal means, due to their high correlation
with the application in which they manifest. Therefore, advances in this field not only help
analyze risk and predict software faults that can lead to failures inside the IT systems of CIs,
but can also help develop a umbrella-like approach for automatically detecting these types of
errors in all software applications.
Experiments also showed that, by using the same methodology, different types of logical
errors could be detected in a wide variety of situations and functionality. The methodology
seems able to detect existing logical vulnerabilities regardless of the type of functionality
implemented inside an application and, also, regardless of the types of machinery handled by
them. Thus, it seems possible to apply this methodology to cover instances of software threats
in known risk assessment methodologies (see §4).
Secondly, critical infrastructures are exposed to external threats that originate from other
infrastructures, which CIs rely on (e.g. a communications infrastructure possibly relies on a
power plant for energy in order to provide its services). Impact of external threats on multiple
CIs is considerable since: (i) related case studies (like the California Blackout Scenario) suggest
that failures in interconnected infrastructures can create a series of cascading failures that may
impact an entire nation in multiple sectors, (ii) CIs heavily depend on each other and (iii)
researchers still have much ground to be covered in order to analyze the risk and propose
solution for these types of failures.
The methodology presented in §5 of this dissertation extends the dependency risk
methodology of Kotzanikolaou et al. (Kotzanikolaou, 2011) (Kotzanikolaou, 2013) by
engaging graph centrality measures to cope with the aforementioned issues concerning
100
infrastructure dependencies. The centrality measures are used as additional criteria to identify
critical infrastructure nodes in a dependency risk graph that significantly affect the critical risk
paths in the graph and are, therefore, good candidates for applying mitigation controls. The
results of the feature selection algorithms confirm that the most critical paths in a dependency
risk graph tend to involve nodes with high centrality values. However, several centrality metrics
are available and they contribute to node selection and overall risk mitigation in various
degrees. For this reason, numerous simulation experiments were conducted to determine the
best metrics and combinations of metrics. Experimental results demonstrate that aggregating
all the centrality sets to identify nodes with high overall centrality values yields a good, and
intuitively appealing, mitigation strategy. Another result is that critical infrastructure nodes
with high closeness and degree centrality values appear to have the highest impact on the overall
risk of a dependency risk graph. If the focus is on mitigating the cumulative risk of a percentage
of the top critical paths in a dependency risk graph, then, in almost every case tested, the
information gain approach proposed in this dissertation is more efficient than manually
choosing nodes. This result holds for all three mitigation mechanisms evaluated in this research.
The single biggest advantage of the proposed risk mitigation algorithm is that it can identify
critical infrastructure nodes that may not be in the top critical path, but still significantly affect
the overall risk in a dependency graph. To our knowledge, this is the first viable implemented
methodology able to pinpoint points-of-failure in a web of interconnected CIs.
Finally, our study shed light on the aspects concerning the evolution of failures across
interconnected CIs. Empirical methods have been criticized for the lack of the required
statistical data needed to assess the likelihood of potential events. While probability data may
be difficult to collect for various CIs, efforts have already been made for specific sectors. For
example, Carreras et al. (Carreras, 2012) demonstrate statistical studies of blackouts which
allow the identification of critical power lines (or groups of power lines) for a given network
model. This provides a technique for identifying critical clusters of lines that are likely to trigger
or propagate cascade effects, due to vulnerabilities of their power lines. The approach presented
in the final chapter of this dissertation also inherits from network-based methods as it combines
a method for discovering dependency risk paths with an automated modeling and analysis tool.
It allows the dependencies of the connected infrastructures to be depicted in a graph and critical
paths to be identified. The tool presented in the final chapter of this dissertation, CIDA, is a
modeling and analysis tool focusing on the study of large-scale dependency scenarios for
proactive analysis. The primary goal of CIDA is to help risk assessors and CIP decision makers
to assess dependency risks proactively, before an initiating threat has been realized. By
analyzing the complete set of the potential dependency paths, the risk assessors may project all
Conclusions
101
the cascading effects that may potentially be realized and thus flag dependency risks above a
threshold that need further attention.
In addition, CIDA can be used to run specific scenarios that may be of particular interest for
the risk assessors. It can provide useful information and reveal “hidden” dependency risks, and
also specifically examine particular realistic scenarios. For example the risk assessors may use
CIDA to examine “what-if” scenarios that only consider initiating security events affecting one
(or some) nodes. CIDA may also be used as an efficient tool for the proactive assessment of
risk mitigation controls and therefore, increasing resilience. Based on real input data, it is
feasible to examine hundreds of variant scenarios, even past incidents.
6.2 Publications
Our contribution is published in peer-reviewed journals, conferences and book chapters,
namely:
Publications in peer-reviewed, academic journals:
j1. George Stergiopoulos., Panayiotis Kotzanikolaou, Marianthi Theoharidou, Dimitris
Gritzalis, "Risk mitigation strategies for Critical Infrastructures based on graph centrality
analysis", International Journal of Critical Infrastructure Protection, Elsevier, July 2015.
j2. George Stergiopoulos, Panayiotis Katsaros, Dimitris Gritzalis “Detecting logical errors and
race conditions in software using dynamic invariant assertions and symbolic execution”
Submitted to Journal of Information and Software Technology (JoIST), Elsevier, in July
2015.
j3. George Stergiopoulos., Panayiotis Kotzanikolaou, Marianthi Theoharidou, Dimitris
Gritzalis, "Time-based Critical Infrastructure Dependency Analysis for Large-Scale and
Cross-Sectoral Failures" Submitted to International Journal of Critical Infrastructure
Protection, Elsevier, in June 2015.
Publications in peer-reviewed, international conferences:
c1. George Stergiopoulos., Panayiotis Kotzanikolaou, Marianthi Theoharidou, Dimitris
Gritzalis, “Using centrality metrics in CI dependency risk graphs for efficient risk
mitigation”, in Proc. of the 9th IFIP International Conference on Critical Infrastructure
Protection (CIP-2015), Springer, USA, March 2015.
c2. George Stergiopoulos, Panagiotis Petsanas, Panagiotis Katsaros, Dimitris Gritzalis,
“Automated exploit detection using path profiling: The disposition should matter, not the
position”, in Proc. of the 12th International Conference on Security and Cryptography
(SECRYPT-2015), pp. 100-111, ScitePress, France, July 2015 .
c3. George Stergiopoulos, Marianthi Theoharidou, Dimitris Gritzalis, "Using logical error
detection in Remote-Terminal Units to predict initiating events of Critical Infrastructures
102
failures", in Proc. of the 3rd International Conference on Human Aspects of Information
Security, Privacy and Trust (HCI-2015), Springer, USA, August 2015
c4. George Stergiopoulos, Panagiotis Katsaros, Dimitris Gritzalis, “Automated detection of
logical errors in programs”, in Proc. of the 9th International Conference on Risks and
Security of Internet and Systems (CRiSIS-2014), Springer, August 2014
c5. George Stergiopoulos, Panagiotis Katsaros, Dimitris Gritzalis, “Source code profiling and
classification for automated detection of logical errors”, in Proc. of the 3rd International
Seminar on Program Verification, Automated Debugging and Symbolic Computation
(PAS-2014), Austria, July 2014
c6. George Stergiopoulos, Vasilis Tsoumas, Dimitris Gritzalis, "On Business Logic
Vulnerabilities Hunting: The APP_LogGIC Framework", in Proc. of the 7th International
Conference on Network and System Security (NSS 2013), pp. 236-249, Springer (LNCS
7873), June 2013.
c7. George Stergiopoulos, Bill Tsoumas, Dimitris Gritzalis, “Hunting application-level logical
errors”, in Proc. of the 4th International Symposium on Engineering Secure Software and
Systems (ESSOS-2012), pp. 135-142, Springer (LNCS 7159), February 2012.
Publications in peer-reviewed book chapters:
b1. George Stergiopoulos, Marianthi Theoharidou, Panagiotis Kotzanikolaou, Dimitris
Gritzalis, "Using centrality measures in dependency risk graphs for efficient risk
mitigation", in Critical Infrastructure Protection IX, Shenoi S. (Ed.), pp. 25-40, Springer,
2015.
As a PhD candidate Mr. Stergiopoulos was involved in other research in the field of
Information Security and authored or co-authored the following three peer-reviewed conference
papers, i.e.:
c8. Stergiopoulos G., Kandias M., Gritzalis D., "Approaching Encryption through Complex
Number Logarithms" (position paper), in Proc. of the 10th International Conference on
Security and Cryptography (SECRYPT-2013), pp. 574-579, Samarati P., et al. (Eds.),
Iceland, July 2013
c9. Faily S., Stergiopoulos G., Katos V., Gritzalis D., "Water, water, everywhere: Nuances for
a Water Industry Critical Infrastructure specification exemplar", in Proc. of the 10th
International Conference on Critical Infrastructures Security (CRITIS-2015), Germany,
October 2015 (to appear).
Conclusions
103
c10. Gritzalis D., Stavrou V., Kandias M., Stergiopoulos G., “Insider Threat: Εnhancing BPM
through Social Media”, in Proc. of the 6th IFIP International Conference on New
Technologies, Mobility and Security (NMTS-2014), Springer, UAE, April 2014.
6.3 Future work
Critical Infrastructures are the backbone of any nation. Securing them is a tedious and
complex process. While considerable effort has been made towards this goal, even more after
various events that had global consequences like the Stuxnet attack, the California Blackout
Scenario etc., still much work needs to be done towards effectively securing CIs from all types
of threats. As discussed in §1 and §2, new attack vectors emerge as CIs become more and more
interconnected and technology advances. This fact introduces challenges in critical
infrastructure and information systems security; challenges that carry along opportunities for
further research.
Firstly, as discussed in §4, the technology behind automatically detecting logical errors is
still in an early stage. Future work can focus on extending the class of application logic
vulnerabilities that can identified. In addition, we plan to extend this methodology to deal with
diverse programming languages and business logic constructs. This will require creating input
models of the functionality in multiple software applications. We plan to extend this work to
involve semantic constructs such as XBRL or OWL semantic constructs. This effort will allow
us to apply PLATO to a larger set of applications, since modeling complex constructs of
functionality into mathematical equations will allow a complex modelling of source code
functionality using dynamic analysis.
As a result, future work can explore any differences logical errors and the types of flaws that
they produce, thus effectively mapping potential failures and threats in Critical infrastructures
into specific software functionality. For example, future work can examine whether programs
sending PLC instructions through MODBUS protocols rely on specific security restrictions to
avoid potential failures due to human-error or not, involving analysis of the initial PLC
firmware and the controls applied on it.
In §5 this dissertation proposed risk mitigation mechanisms and ways to pinpoint dangerous
CI nodes in risk dependency graphs. These mitigation algorithms are rather restrictive since
they do not taker into consideration potential geographical or national restrictions that might
apply on various CI nodes in a given dependency graph. An obvious direction of future work
is to mount a GIS framework utilizing latitude and longtitude details for all given CIs and
outsource information gathering on these matters on various available tools like Google Maps.
Another possible direction of research is the examination of the possibility that the use
dependency analysis in critical infrastructures augments threat analysis in risk assessments.
Currently, risk assessment methodologies scarcely take into consideration external threats due
104
to interdependencies. Utilizing the proposed methodologies during Risk Assessment may
augment the threat model of each assessment and help pinpoint new threats and potential
instabilities.
Moreover, the analysis of the cascading failures amongst interdepended critical
infrastructures in §6 does not take into consideration formal risk assessment output or ISO
standards. An obvious direction for future work is to analyze and include results from risk
assessment reports into these security models for analyzing webs of interconnected CIs.
Similarly, future work can include a dynamic implementation of CIs that enter the dependency
graph and being able to recalibrate impact assessment on-the-fly.
Future work can also implement a risk rank and corresponding output from the proposed
methodology (§6) that can reference to specific security controls tied to external CI threats; like
proposing specific ISO-certified countermeasures for specific types of threats (e.g CIDA could
propose the utilization of extra UPS and alternate power support for a given CI if there is a
high-impact-high-evolution external threat from a power plant to the aforementioned
infrastructure). Future research should also focus on aspects that have not been measured in our
experiments and remain yet unclear. For instance, clarifying the differences between different
types of interconnections between CIs, and the effect these have on cascading failures.
Overall, results in this dissertation provide evidence that critical infrastrcutures require multi-
level protection against threats that stem from multiple input vector and in various granularity
levels. Developing methodologies to protect CIs both in software-level interaction of users with
machines and also on the uncharted area of external interdependencies seems to be of utmost
importance. In this context, it would be prudent to extend the common body of knowledge for
the security domain (e.g. the one described in (Theoharidou and Gritzalis, 2007; Gritzalis et al.,
2005)) to include the aforementioned aspects of security (such as impact of authorization
decisions via software interations, unique attacks, adoption of security controls, etc.).
6.4 Concluding remarks
(Inter)dependencies between critical infrastructures are a key factor in critical infrastructure
protection because they may allow a failure that is seemingly isolated in one critical
infrastructure to cascade to multiple critical infrastructures. The protection of such
infrastructures is crucial, as CIs constantly merge with a person’s everyday life and support all
aspects of a nation. This work explored the current state of protection of CIs in two different
granularity levels; that of software-based logical vulnerabilities and of interdependencies
between infrastructures.
Conclusions
105
This thesis utilized the lessons learned while studying the interdependencies between critical
infrastructures together with analyzing software for the sole purpose of finding a way to secure
it against initiating serious failures in infrastructure systems. It presented how logical
vulnerabilities in software can - and under which circumstances- be detected using automated
analysis. On the broader scale, it introduced risk mitigation mechanisms for use in complex
infrastructure dependency risk graphs and developed a tool able to predict, analyze and
visualize the impact of cascading failures between multiple critical infrastructures.
106
(this page is intentionally left blank)
Appendix
107
Appendix
A. Time-Based Analysis of cascading failures using the CIDA tool
A1. Tool libraries and development
This wiki is used as a reference to documentation and data used by CIDA. It is intended for
users trying to work and understand the methodology behind CIDA. The CIDA tool utilizes (i)
the Blueprints collection and, especially, the Neo4J graph library as a technology to realize its
dependency analysis modeling methodology, and (ii) the jFuzzyLogic Java library to implement
its fuzzy logic control system.
Neo4J builds upon the property graph model; nodes may have various labels and each label
can serve as an informational entity. The nodes are connected via directed, typed relationships.
The Blueprints collection can be found here: https://github.com/tinkerpop/blueprints/wiki.
The Neo4J project can be found here: http://neo4j.com/developer/get-started/.
jFuzzyLogic implements Fuzzy Control Language (FCL) which standardizes programming
fuzzy logic systems and reduces programming times by removing the need to perform to boiler
plate programming tasks. The jFuzzyLogic can be found here:
http://jfuzzylogic.sourceforge.net/html/index.html.
Appendix
108
A2. Fuzzy Logic IF THEN rules
Fuzzy sets along with linguistic IF-THEN rules are used to calculate the fuzzy value of the
expected time-based impact value. Rules are usually expressed in the form: IF variable IS property
THEN action.
Initially a processing stage invokes the appropriate IF-THEN rules and generates a result for
each rule. Then these results are combined to output a set of truth values. Each IF-THEN result is,
essentially, a membership function and truth value controlling the output set, i.e. the linguistic
variables impact and time.
Membership sets are dynamically created for each Critical Infrastructure during CIDA's
analysis. Yet, the IF-THEN rules used by CIDA to combine fuzzy sets and calculate output are
the following:
RULEBLOCK No1
// Use 'min' for 'and' (also implicit use 'max'
// for 'or' to fulfill DeMorgan's Law)
AND : MIN;
// Use 'min' activation method
ACT : MIN;
// Use 'max' accumulation method
ACCU : MAX;
`RULE 1 : IF Impact IS Very_High AND Time IS Early THEN Impact_T
is Medium;`
`RULE 2 : IF Impact IS Very_High AND Time IS Medium THEN Impact_T
is High;`
`RULE 3 : IF Impact IS Very_High AND Time IS Late THEN Impact_T
is Very_High;`
`RULE 4 : IF Impact IS Very_High AND Time IS Very_Late THEN
Impact_T is Very_High;`
`RULE 5 : IF Impact IS High AND Time IS Early THEN Impact_T is
Medium;`
`RULE 6 : IF Impact IS High AND Time IS Medium THEN Impact_T is
Medium;`
`RULE 7 : IF Impact IS High AND Time IS Late THEN Impact_T is
High;`
Appendix
109
`RULE 8 : IF Impact IS High AND Time IS Very_Late THEN Impact_T
is High;`
`RULE 9 : IF Impact IS Medium AND Time IS Early THEN Impact_T is
Low;`
`RULE 10 : IF Impact IS Medium AND Time IS Medium THEN Impact_T
is Medium;`
`RULE 11 : IF Impact IS Medium AND Time IS Late THEN Impact_T is
Medium;`
`RULE 12 : IF Impact IS Medium AND Time IS Very_Late THEN Impact_T
is Medium;`
`RULE 13 : IF Impact IS Low AND Time IS Early THEN Impact_T is
Very_Low;`
`RULE 14 : IF Impact IS Low AND Time IS Medium THEN Impact_T is
Very_Low;`
`RULE 15 : IF Impact IS Low AND Time IS Late THEN Impact_T is
Low;`
`RULE 16 : IF Impact IS Low AND Time IS Very_Late THEN Impact_T
is Low;`
`RULE 17 : IF Impact IS Very_Low AND Time IS Early THEN Impact_T
is Very_Low;`
`RULE 18 : IF Impact IS Very_Low AND Time IS Medium THEN Impact_T
is Very_Low;`
`RULE 19 : IF Impact IS Very_Low AND Time IS Late THEN Impact_T
is Very_Low;`
`RULE 20 : IF Impact IS Very_Low AND Time IS Very_Late THEN
Impact_T is Very_Low;`
END_RULEBLOCK
A3. Impact Time Tables
Three growth rates (linear, logarithmic and exponential) are used by CIDA to model the
evolution of impact, when a failure manifests itself in an infrastructure. These rates are used to
Appendix
110
pre-compute all possible values of impact I(t) in specific points t on CIDA's time scale (in min):
T = {15, 60, 180, 720, 1440, 2880, 10080, 20160, 40320, 60480 }.
The yellow line depicts the time point when the worst-case impact I will manifest (maximum
impact value).
Tables below reference all possible combinations of impact, time point and growth rate of a
failure in a Critical Infrastructure (I,T,G). They are used as input to a fuzzy logic ranking system,
to group impact and time values in fuzzy membership sets. Tables are provided below:
Exponential Tables
Appendix
111
Appendix
112
Linear Tables
Appendix
113
Appendix
114
Appendix
115
Logarithmic Tables
Appendix
116
Appendix
117
A3. Guide to installing the CIDA Eclipse project
The CIDA project is using Maven to install libraries and extension, along with those found
under the \lib folder. Make sure your Eclipse instance has Maven support installed. Follow these
steps to import a project to your computer:
1. Create a workspace folder for eclipse (if you don't have one).
2. Clone a project folder to your destination system:
3. The GUI CIDA tool (CIPTIMEFL) or
4. the Random Graph Stress Analyzer (CIP2014)
5. Transfer the folder under your \workspace Eclipse folder.
6. Select File > Import.
7. Select General > Existing project into workspace.
8. Select the project folder (under select root folder).
9. Say Finish.
References
118
References
1. Department of Homeland Security (DHS), What Is Critical Infrastructure?, October 2013,
http://www.dhs.gov/what-critical-infrastructure
2. Moteff, John, Claudia Copeland, and John Fischer. "Critical infrastructures: what makes an
infrastructure critical?." Library of Congress Washington DC Congressional Research Service,
2003.
3. Abel W. Agents, trojans and tags: The next generation of investigators. International Review of
Law, Computers & Technology 2009;23(1-2):99–108.
4. W. W. Peng, D. R. Wallace, Software error analysis, NIST Special Publication 500 (1993) 209.
5. M. Kimura, Software vulnerability: definition, modelling, and practical evaluation for e-mail
transfer software, International journal of pressure vessels and piping 83 (4) (2006) 256–261.
6. Codepro (2015). URL https://developers.google.com/java-dev-tools/codepro/doc/
7. Ucdetector (2015). URL http://www.ucdetector.org/
8. Pmd (2015).URL http://pmd.sourceforge.net/
9. Hovemeyer, W. Pugh, Finding bugs is easy, ACM Sigplan Notices 39 (12) (2004) 92–106.
10. Coverity save audit tool (2015). URL http://www.coverity.com
11. The java pathfinder tool (2015). URL http://babelfish.arc.nasa.gov/trac/jpf/
12. M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, C. Xiao, The
daikon system for dynamic detection of likely invariants, Science of Computer Programming 69
(1) (2007) 35–45.
13. G. Stergiopoulos, B. Tsoumas, D. Gritzalis, Hunting application-level logical errors, in:
Engineering Secure Software and Systems, Springer, 2012, pp. 135–142.
14. C. S. P as areanu, W. Visser, Verification of java programs using symbolic execution and invariant
generation, in: Model Checking Software, Springer, 2004, pp. 164–181.
15. V. Felmetsger, L. Cavedon, C. Kruegel, G. Vigna, Toward automated detection of logic
vulnerabilities in web applications, in: USENIX Security Symposium, 2010, pp. 143–160.
16. CWE, The common weakness enumeration (cwe) community (2015). URL https://cwe.mitre.org/
17. G. Stergiopoulos, B. Tsoumas, D. Gritzalis, On business logic vulnerabilities hunting: The
APP_Loggic framework, in: Network and System Security, Springer, 2013, pp. 236–249.
18. G. Stergiopoulos, P. Katsaros, D. Gritzalis, Automated detection of logical errors in programs, in:
Proc. of the 9th International Conference on Risks & Security of Internet and Systems, 2014.
19. T. Boland, P. E. Black, Juliet 1.1 c/c++ and java test suite, Computer (10) (2012) 88–90.
20. The post 2015 Hyogo Framework for Action: Managing risks to achieve resilience achieve
resilience, European Commission, Brussels, 2014, COM(2014) 216 final.
21. A. Zeller, Isolating cause-effect chains from computer programs, in: Proceedings of the 10th ACM
SIGSOFT symposium on Foundations of software engineering, ACM, 2002, pp. 1–10.
22. X. Zhang, N. Gupta, R. Gupta, Locating faults through automated predicate switching, in:
Proceedings of the 28th international conference on Software engineering, ACM, 2006, pp. 272–
281.
23. M. Weiser, Program slicing, in: Proceedings of the 5th international conference on Software
engineering, IEEE Press, 1981, pp. 439–449.
24. X. Zhang, N. Gupta, R. Gupta, Pruning dynamic slices with confidence, in: ACM SIGPLAN
Notices, Vol. 41, ACM, 2006, pp. 169–180.
25. A. Doupe, B. Boe, C. Kruegel, G. Vigna, Fear the ear: discovering and mitigating execution after
redirect vulnerabilities, in: Proceedings of the 18th ACM conference on Computer and
communications security, ACM, 2011, pp. 251–262.
26. D. Balzarotti, M. Cova, V. V. Felmetsger, G. Vigna, Multi-module vulnerability analysis of web-
based applications, in: Proceedings of the 14th ACM conference on Computer and
communications security, ACM, 2007, pp. 25–35.
27. The daikon invariant detector manual (2015). URL http://groups.csail.mit.edu/pag/daikon/
References
119
28. R. A. Martin, S. Barnum, Common weakness enumeration (CWE) status update, ACM SIGAda
Ada Letters 28 (1) (2008) 88–91.
29. G. K. Baah, Statistical causal analysis for fault localization.
30. E. R. Harold, Java I/O, ” O’Reilly Media, Inc.”, 2006.
31. E. Gosling, B. Joy, G. L. Steele Jr, G. Bracha, A. Buckley, The Java Language Specification,
Pearson Education, 2014.
32. Java platform, standard edition 7 api specification (2015). URL
http://docs.oracle.com/javase/7/docs/api/
33. G. Stoneburner, A. Goguen, A. Feringa, Risk management guide for information technology
systems, Nist special publication 800 (30) (2002) 800–30.
34. E. Perfilieva, J. Moˇckoˇr, Mathematical principles of fuzzy logic, Springer Science & Business
Media, 1999.
35. G. Albaum, The likert scale revisited, Journal-Market research society 39 (1997) 331–348.
36. N. A. Abramson, Introduction to Information Theory and Coding, McGraw Hill, 1964.
37. H. Etzkorn, C. G. Davis, Automatically identifying reusable oo legacy code, Computer 30 (10)
(1997) 66–71.
38. Y. Yang, J. O. Pedersen, A comparative study on feature selection in text categorization, in: ICML,
Vol. 97, 1997, pp. 412–420.
39. E. J. Glover, G. W. Flake, S. Lawrence, W. P. Birmingham, A. Kruger, C. L. Giles, D. M. Pennock,
Improving category specific web search by learning query modifications, in: Applications and the
Internet, 2001. Proceedings. 2001 Symposium on, IEEE, 2001, pp. 23–32.
40. S. Ugurel, R. Krovetz, C. L. Giles, What’s the code?: automatic classification of source code
archives, in: Proceedings of the eighth ACM SIGKDD international conference on Knowledge
discovery and data mining, ACM, 2002, pp. 632–638.
41. NVD, National vulnerability database, [online] http://nvd.nist.gov (2015). URL
http://nvd.nist.gov
42. R. Jhala, R. Majumdar, Software model checking, ACM Comput. Surv. 41 (4) (2009) 21:1–21:54.
doi:10.1145/1592434.1592438. URL http://doi.acm.org/10.1145/1592434.1592438
43. P. Chhabra, L. Bansal, An effective implementation of improved halstead metrics for software
parameters analysis (2014).
44. E. Bray, K. Brune, D. A. Fisher, J. Foreman, M. Gerken, C4 software technology reference guide-
a prototype., Tech. rep., DTIC Document (1997).
45. G. K. Gill, C. F. Kemerer, Cyclomatic complexity density and software maintenance productivity,
Software Engineering, IEEE Transactions on 17 (12) (1991) 1284–1288.
46. W. J. Hansen, Measurement of program complexity by the pair:(cyclomatic number, operator
count), ACM SIGPLan Notices 13 (3) (1978) 29–33.
47. E. Rosenberg, T. Hammer, Metrics for quality assurance and risk assessment, Proc. Eleventh
International Software Quality Week, San Francisco, CA.
48. Using code quality metrics in management of outsourced development and maintenance (2015).
URL http://www.mccabe.com/pdf/McCabeCodeQualityMetrics-OutsourcedDev.pdf
49. Foundations of fuzzy logic, fuzzy operators (2015). URL
www.mathworks.com/help/toolbox/fuzzy/bp78l6_-1.html
50. W. Van Leekwijck, E. E. Kerre, Defuzzification: criteria and classification, Fuzzy sets and
systems 108 (2) (1999) 159–178.
51. P. Cingolani, J. Alcala-Fdez, jfuzzylogic: a robust and flexible fuzzy- logic inference system
language implementation., in: FUZZ-IEEE, Citeseer, 2012, pp. 1–8.
52. H. K. Wright, M. Kim, D. E. Perry, Validity concerns in software engineering research, in:
Proceedings of the FSE/SDP workshop on Future of software engineering research, ACM, 2010,
pp. 411–414.
53. NSA, On Analyzing Static Analysis Tools, National Security Agency, 2011.
54. NSA,Static Analysis Tool Study-Methodology, National Security Agency, 2012.
55. G. Rothermel, S. Elbaum, A. Kinneer, H. Do, Software-artifact infrastructure repository (2006).
References
120
56. H. Do, S. Elbaum, G. Rothermel, Supporting controlled experimentation with testing techniques:
An infrastructure and its potential impact, Empirical Software Engineering 10 (4) (2005) 405–
435.
57. V. Okun, A. Delaitre, P. E. Black, Report on the static analysis tool exposition (sate) iv, NIST
Special Publication 500 (2013) 297.
58. W. D., Jamod java modbus implementation, http://jamod.sourceforge.net/ (2015). URL
http://jamod.sourceforge.net/
59. Flowserve l75 series electric actuator, fcd lmaim7502-00, http://jamod.sourceforge.net/ (July
2005). URL http://jamod.sourceforge.net/
60. PLCSimulator, Modbus plc simulator, http://www.plcsimulator.org/ (2014). URL
http://www.plcsimulator.org/
61. R. Natella, D. Cotroneo, J. A. Duraes, H. S. Madeira, On fault representativeness of software fault
injection, Software Engineering, IEEE Transactions on 39 (1) (2013) 80–96.
62. N. Rutar, C. B. Almazan, J. S. Foster, A comparison of bug finding tools for java, in: Software
Reliability Engineering, 2004. ISSRE 2004. 15th International Symposium on, IEEE, 2004, pp.
245–256.
63. Krutz, R.: Securing SCADA Systems. Wiley, Indianapolis (2005)
64. Alcaraz,C. ,Lopez,J. ,Zhou,J. ,Roman,R.: Secure SCADA framework for the protection of energy
control systems. Concurrency Comput. Pract. Experience 23(12), 1414–1430 (2011)
65. P. Kotzanikolaou, M. Theoharidou, D. Gritzalis, Cascading effects of common-cause failures in
critical infrastructures, in: J. Butts, S. Shenoi (Eds.), Critical Infrastructure Protection, Vol. 417
of IFIP Advances in Information and Communication Technology, Springer, 2013, pp. 171–182.
66. P. Kotzanikolaou, M. Theoharidou, D. Gritzalis, Assessing n-order dependencies between critical
infrastructures, IJCIS 9 (1/2) (2013) 93–110.
67. P. Kotzanikolaou, M. Theoharidou, D. Gritzalis, Interdependencies between critical
infrastructures: Analyzing the risk of cascading effects, in: Critical Information Infrastructure
Security, Springer, 2013, pp. 104–115.
68. M. Theoharidou, P. Kotzanikolaou, D. Gritzalis, Risk assessment methodology for interdependent
critical infrastructures, International Journal of Risk Assessment and Management 15 (2) (2011)
128–148.
69. M. Theoharidou, P. Kotzanikolaou, D. Gritzalis, A multi-layer criticality assessment methodology
based on interdependencies, Computers & Security 29 (6) (2010) 643–658.
70. The post 2015 Hyogo Framework for Action: Managing risks to achieve resilience achieve
resilience, European Commission, Brussels, 2014, COM(2014) 216 final.
71. NIPP 2013: Partnering for Critical Infrastructure Security and Resilience, Dept. of Homeland
Security, 2013.
72. P. Bonacich, Power and centrality: A family of measures, American journal of sociology (1987)
1170–1182.
73. Microsoft, Msdn article (2015). URL https://msdn.microsoft.com/en-us/library/ms175382.aspx
74. Y. Yang, J. Pederson, A comparative study on feature selection in text categorization, in:
Fourteenth International Conference on Machine Learning (ICML’97), 1997, pp. 412–420.
75. N. Abramson, Information Theory and Coding, McGraw-Hil, 1963.
76. E. J. Glover, G. W. Flake, S. Lawrence, W. P. Birmingham, A. Kruger, L. C. Giles, D. M. Pennoek,
Improving category specific web search by learning query modification, in: Symposium on
Applications and the Internet (SAINT), 2001 IEEE, 2001, pp. 23–31.
77. E. Ugurel, R. Krovetz, C. L. Giles, D. M. Pennock, E. J. Glover, H. Zha, What’s the code?:
automatic classification of source code archives, in: SIGKDD international conference on
Knowledge discovery and data mining (ACM KDD), 8th ACM, 2002, pp. 632–638.
78. G. Karegowda, A. Manjunath, M. Jayaram, Comparative study of attribute selection using gain
ratio and correlation based feature selection, International Journal of Information Technology and
Knowledge Management 2 (2) (2010) 271–277.
References
121
79. E. Luiijf, A. Nieuwenhuijs, M. Klaver, M. van Eeten, E. Cruz, Empirical findings on critical
infrastructure dependencies in Europe, in: Critical Information Infrastructure Security, Springer,
2009, pp. 302–310.
80. P. Bonacich, Power and centrality: A family of measures, American journal of sociology (1987)
1170–1182.
81. E. Jouili, V. Vansteenberghe, An empirical comparison of graph databases, in: Social Computing
(SocialCom), 2013 International Conference on, 2013, pp. 708–715.
doi:10.1109/SocialCom.2013.106. URL
http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=6693403
82. B. Shao, H. Wang, Y. Xiao, Managing and mining large graphs: systems and implementations, in:
Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, ACM,
2012, pp. 589–592.
83. Rinaldi, J. Peerenboom, T. Kelly, Identifying, understanding, and analyzing critical infrastructure
interdependencies, IEEE Control Systems Magazine 21 (6) (2001).
84. R. Kozik, M. Choras, Current cyber security threats and challenges in critical infrastructures
protection, in: Informatics and Applications (ICIA),2013 Second International Conference on,
2013, pp. 93–97.
85. Usov, C. Beyel, E. Rome, U. Beyer, E. Castorini, P. Palazzari, A. Tofani, The DIESIS approach
to semantically interoperable federated critical infrastructure simulation, in: Advances in System
Simulation (SIMUL), 2010 Second International Conference on, IEEE, 2010, pp. 121–128.
86. E. Rome, S. Bologna, E. Gelenbe, E. H. Luiijf, V. Masucci, DIESIS: an interoperable european
federated simulation network for critical infrastructures, in: Proceedings of the 2009 SISO
European Simulation Interoperability Workshop, Society for Modeling & Simulation
International, 2009, pp. 139–146.
87. J. R. Marti, J. A. Hollman, C. Ventura, J. Jatskevich, Dynamic recovery of critical infrastructures:
real-time temporal coordination, International Journal of Critical Infrastructures 4 (1) (2008) 17–
31.
88. Critical infrastructure preparedness and resilience research network (2014). URL
http://www.ciprnet.eu/
89. M. Ouyang, Review on modeling and simulation of interdependent critical infrastructure systems,
Rel. Eng. & Sys. Safety 121 (2014) 43–60.
90. G. Andersson, P. Donalek, R. Farmer, N. Hatziargyriou, I. Kamwa, P. Kundur, N. Martins, J.
Paserba, P. Pourbeik, J. Sanchez-Gasca, R. Schulz, A. Stankovic, C. Taylor, V. Vittal, Causes of
the 2003 major grid blackouts in North America and Europe, and recommended means to improve
system dynamic performance, IEEE Transactions on Power Systems 20 (4) (2005) 1922–1928.
91. S. V. Buldyrev, R. Parshani, G. Paul, H. E. Stanley, S. Havlin, Catastrophic cascade of failures in
interdependent networks, Nature 464 (2010) 10251028.
92. S. Panzieri, R. Setola, Failures propagation in critical interdependent infrastructures, International
Journal of Modelling, Identification and Control 3 (1) (2008) 69–78.
93. A. Vespignani, Complex networks: The fragility of interdependenc, Nature 464 (2010) 984985.
94. D. Zhou, A. Bashan, Y. Berezin, R. Cohen, S. Havlin, On the dynamics of cascading failures in
interdependent networks, arXiv preprint arXiv:1211.2330.
95. L. Duenas-Osorio, S. M. Vemuru, Cascading failures in complex infrastructure systems, Structural
safety 31 (2) (2009) 157–167.
96. E. Zio, G. Sansavini, Modeling interdependent network systems for identifying cascade-safe
operating margins, Reliability, IEEE Transactions on 60 (1) (2011) 94–101.
97. M. Van Eeten, A. Nieuwenhuijs, E. Luiijf, M. Klaver, E. Cruz, The state and the threat of
cascading failure across critical infrastructures: the implications of empirical evidence from media
incident reports, Public Administration 89 (2) (2011) 381–400.
98. L. Franchina, M. Carbonelli, L. Gratta, M. Crisci, An impact-based approach for the analysis of
cascading effects in critical infrastructures, International journal of critical infrastructures 7 (1)
(2011) 73–90.
References
122
99. B. Robert, A method for the study of cascading effects within lifeline networks, International
journal of critical infrastructures 1 (1) (2004) 86–99.
100. B. Utne, P. Hokstad, J. Vatn, A method for risk modeling of interdependencies in critical
infrastructures, Reliability Engineering & System Safety 96 (6) (2011) 671–678.
101. G. H. Kjølle, I. B. Utne, O. Gjerde, Risk analysis of critical infrastructures emphasizing electricity
supply and interdependencies, Reliability Engineering & System Safety 105 (2012) 80–89.
102. P. Kotzanikolaou, M. Theoharidou, D. Gritzalis, Interdependencies between critical
infrastructures: Analyzing the risk of cascading effects, in: S. Bologna, B. Hammerli, D. Gritzalis,
S. Wolthusen (Eds.), CRITIS, Vol. 6983 of Lecture Notes in Computer Science, Springer, 2011,
pp. 104– 115.
103. B. Carreras, D. Newman, I. Dobson, Determining the vulnerabilities of the power transmission
system, in: System Science (HICSS), 2012 45th Hawaii International Conference on, 2012, pp.
2044–2053. doi:10.1109/HICSS.2012.208.
104. D. Judi, A. Kalyanapu, S. Burian, B. Daniel, T. McPherson, Wide-area flood inundation and
infrastructure risk assessment simulation framework, in: Proceedings of the Second IASTED
International Conference on Water Resources Management, 2007.
105. H. Luiijf, D. Stolk, An international tabletop exercise on critical infrastructure protection: the
lessons identified, International journal of critical infrastructures 6 (3) (2010) 293–303.
106. G. Stergiopoulos, P. Kotzanikolaou, M. Theocharidou, D. Gritzalis, CIDA: Critical Infrastructure
Dependency Analysis tool, September 2014). URL https://github.com/geostergiop/CIDA
107. Presidential Policy Directive - Critical Infrastructure Security and Resilience (PPD-21), The
White House, 2013.
108. R. Francis, B. Bekera, A metric and frameworks for resilience analysis of engineered and
infrastructure systems, Rel. Eng. & Sys. Safety 121 (2014) 90–103.
109. W. Kroger, Emerging risks related to large-scale engineered systems, Tech. rep., International
Risk Governance Council (2010).
110. C. Vicknair, M. Macias, Z. Zhao, X. Nan, Y. Chen, D. Wilkins, A comparison of a graph database
and a relational database: a data provenance perspective, in: Proceedings of the 48th annual
Southeast regional conference, ACM, 2010, p. 42.
111. Neo4J graph database (2014). URL http://www.neo4j.org/
112. S. Batra, C. Tyagi, Comparative analysis of relational and graph databases, International Journal
of Soft Computing and Engineering (IJSCE) 2 (2) (2012) 509–512.
113. Green Paper on a european programme for critical infrastructure protection, Commission of the
European Communities, 2005, COM(2005) 576 final.
114. W. Kroger, E. Zio, Vulnerable systems, Springer, 2011.
115. E. E. Lee, J. E. Mitchell, W. A. Wallace, Restoration of services in interdependent infrastructure
systems: A network flows approach, Systems, Man, and Cybernetics, Part C: Applications and
Reviews, IEEE Transactions on 37 (6) (2007) 1303–1317.
116. N. K. Svendsen, S. D. Wolthusen, Connectivity models of interdependency in mixed-type critical
infrastructure networks, Information Security Technical Report 12 (1) (2007) 44–55.
117. N. K. Svendsen, S. D. Wolthusen, Analysis and statistical properties of critical infrastructure
interdependency multi-flow models, in: Information Assurance and Security Workshop, 2007.
IAW’07. IEEE SMC, IEEE, 2007, pp. 247–254.
118. M. Ouyang, L. Duenas-Osorio, An approach to design interface topologies across interdependent
urban infrastructure systems, Reliability Engineering & System Safety 96 (11) (2011) 1462–1473.
119. E. Rosato, L. Issacharoff, F. Tiriticco, S. Meloni, S. Porcellinis, R. Setola, Modelling
interdependent infrastructures using interacting dynamical models, International Journal of
Critical Infrastructures 4 (1) (2008) 63–79.
120. J. Talsma, B. P. BECKER, Q. Gao, E. RUIJGH, Coupling of multiple channel flow models with
openmi, in: 10th International Conference on Hydro-informatics HIC, 2012.
References
123
121. C. Siaterlis, B. Genge, M. Hohenadel, EPIC: A testbed for scientifically rigorous cyber-physical
security experimentation, IEEE Transactions on Emerging Topics in Computing 1 (2) (2013) 319–
330. doi:10.1109/TETC.2013.2287188.
122. Y. Y. Haimes, P. Jiang, Leontief-based model of risk in complex interconnected infrastructures,
Journal of Infrastructure systems 7 (1) (2001) 1–12.
123. J. R. Santos, Y. Y. Haimes, Modeling the demand reduction input-output (i-o) inoperability due
to terrorism of interconnected infrastructures*, Risk Analysis 24 (6) (2004) 1437–1451.
124. J. R. Santos, Inoperability input-output modeling of disruptions to interdependent economic
systems, Systems Engineering 9 (1) (2006) 20–34.
125. R. Setola, S. De Porcellinis, M. Sforna, Critical infrastructure dependency assessment using the
input–output inoperability model, International Journal of Critical Infrastructure Protection 2 (4)
(2009) 170–178.
126. M. A. Ehlen, A. J. Scholand, Modeling interdependencies between power and economic sectors
using the n-able agent-based model, in: Proceedings of the IEEE power engineering society
general meeting, 2005, pp. 2842– 2846.
127. B. Bush, L. Dauelsberg, R. LeClaire, D. Powell, S. Deland, M. Samsa, Critical infrastructure
protection decision support system (cip/dss) project overview, in: Proceedings of the 23rd
international conference of the system dynamics society, 2005, pp. 17–21.
128. S. H. Conrad, R. J. LeClaire, G. P. O’Reilly, H. Uzunalioglu, Critical naional infrastructure
reliability modeling and analysis, Bell Labs Technical Journal 11 (3) (2006) 57–71.
129. S. Walsh, S. Cherry, L. Roybal, Critical infrastructure modeling: An approach to characterizing
interdependencies of complex networks & control systems, in: Human System Interactions, 2009.
HSI ’09. 2nd Conference on, 2009, pp. 637–641. doi:10.1109/HSI.2009.5091052.
130. Gordon K., Dion M., Protection of Critical Infrastructures and the role of Investment Policies
relating to national security, OECD, May 2008.
131. Emergency Management Australia, Critical Infrastructure Emergency Risk Management and
Assurance Handbook, January 2003.
132. Ministry of the Interior and Kingdom Relations, National Risk Assessment Method Guide 2008,
National Security Programme, The Netherlands, June 2008.
133. Insight Consulting, CRAMM User Guide, Issue 5.1 July 2005.
134. ISO/IEC, Information technology — Security techniques — Information security management
systems — Overview and vocabulary ISO/ IEC 27000:2014.
135. P. Pederson, D. Dudenhoeffer, M. Permann and S. Hartley, “Critical Infrastructure
Interdependency Modeling: A Survey of U.S. and International Research”, Idaho National
Laboratory, August 2006
Recommended
![Dams and Energy Sectors Interdependency Study · [Type text] Dams and Energy Sectors Interdependency Study September 2011](https://img.dokumen.tips/doc/110x75/5aebfe197f8b9a45568dcbf7/dams-and-energy-sectors-interdependency-type-text-dams-and-energy-sectors-interdependency.jpg)
