SmartResilience - D5.2: Stress-test framework · 2018-03-23 · specific stresstest scenarios, which will be carried out as an event (- workshop, table top simulation, game, or drill),

PUBLIC DELIVERABLE

H2020 Project: Smart Resilience Indicators for Smart Critical Infrastructure

D5.2 - Stress-test and evaluation framework

Coordinator: Aleksandar Jovanovic EU-VRi Project Manager: Bastien Caillard EU-VRi

European Virtual Institute for Integrated Risk Management Haus der Wirtschaft, Willi-Bleicher-Straße 19, 70174 Stuttgart

Contact: [email protected]

SMART RESILIENCE INDICATORS FOR SMART CRITICAL INFRASTRUCTURES

© 2016-2019 This document and its content are the property of the SmartResilience Consortium. All rights relevant to this document are determined by the applicable laws. Access to this document does not grant any right or license on the document or its contents. This document or its contents are not to be used or treated in any manner inconsistent with the rights or interests of the SmartResilience Consortium or the Partners detriment and are not to be disclosed externally without prior written consent from the SmartResilience Partners. Each SmartResilience Partner may use this document in conformity with the SmartResilience Consortium Grant Agreement provisions. The research leading to these results has received funding from the European Union’s Horizon 2020 Research and Innovation Programme, under the Grant Agreement No 700621.

The views and opinions in this document are solely those of the authors and contributors, not those of the European Commission.

Stress-test and evaluation framework

Report Title: Stress-test and Evaluation Framework

Author(s):

SZŐKÉNÉ PELIKÁN Margit, SZÉKELY Zoltán, MACSÁRI István, JAMBRIK Rudolf, Johan M. SANNE, Magnus RAHMBERG, Hanna MATSCHKE EKHOLM, Alexander S. JOVANOVIC, Amrita CHOUDHARY, Bastien CAILLARD, Peter KLIMEK, Ruben ROQUE, Thomas KNAPE, Gerard DESMOND, Milos JOVANOVIC, Mary-Ellen LANG, Kimberley CAMPBEL, Paul YOUNG, Dmitrij BEZRUKOV, Mirjana NIKOLIC, Dragana BLAZEVIC, AUERKARI Pertti, KOIVISTO Raija, MOLARIUS Ritta, ÁRVAI László, PERÉNYI Dénes, CSAPÓ Gábor

Responsible Project Partner:

BZN Contributing Project Partners:

HNP, R-Tech, IVL, SwissRe, SINTEF, FhG-INT, EU-VRi, SWH, SRH, IBM, MUW

Document data:

File name / Release:

D5.2-Stress-test-framework_v41bc07032018 Release No.: 6

Pages: 111 No. of annexes: 4

Status: Final Dissemination level: PU

Project title: SmartResilience: Smart Resilience Indicators for Smart Critical Infrastructures

Grant Agreement No.: 700621

Project No.: 12135

WP title: WP5: SCI-Resilience: Application in Smart City Case Studies

Deliverable No: D5.2

Date: Due date: March 7, 2018 Submission date: March 7, 2018

Keywords: Stress-test, evaluation, framework, case-studies

Reviewed by: E. Bellini Review date: November 20, 2017

D. Verner Review date: November 17, 2017

Approved by Coordinator:

A. Jovanović Approval date: March 7, 2018

Budapest, March 2018

Ref. Ares(2018)1276440 - 07/03/2018

SmartResilience: Indicators for Smart Critical Infrastructures

page i

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Release History

Release No.

Date Description / Change

1 August 26, 2017 Initial draft

2 October 12, 2017 Changes according to discussions during WPL meeting on September 15, 2017 in Brussels, e.g. stress-test of SCIs using the functionality assessment methodology (T3.3).

3 November 1, 2017 Functionality level and threshold definition inputs from partners added.

4 November 9, 2017 Release candidate 1

5 November 22, 2017 Final version

6 March 7, 2018 Revised version submitted to the European Commission


page ii

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Project Contact

EU-VRi – European Virtual Institute for Integrated Risk Management Haus der Wirtschaft, Willi-Bleicher-Straße 19, 70174 Stuttgart, Germany Visiting/Mailing address: Lange Str. 54, 70174 Stuttgart, Germany Tel: +49 711 410041 27, Fax: +49 711 410041 24 – www.eu-vri.eu – [email protected] Registered in Stuttgart, Germany under HRA 720578

SmartResilience Project

Modern critical infrastructures are becoming increasingly smarter (e.g. the smart cities). Making the infrastructures smarter usually means making them smarter in the normal operation and use: more adaptive, more intelligent etc. But will these smart critical infrastructures (SCIs) behave smartly and be smartly resilient also when exposed to extreme threats, such as extreme weather disasters or terrorist attacks? If making existing infrastructure smarter is achieved by making it more complex, would it also make it more vulnerable? Would this affect resilience of an SCI as its ability to anticipate, prepare for, adapt and withstand, respond to, and recover? What are the resilience indicators (RIs) which one has to look at?

These are the main questions tackled by SmartResilience project.

The project envisages answering the above questions in several steps: (#1) By identifying existing indicators suitable for assessing resilience of SCIs, (#2) By identifying new smart resilience indicators including those from Big Data, (#3) By developing, a new advanced resilience assessment methodology based on smart RIs and the resilience indicators cube, including the resilience matrix, (#4) By developing the interactive SCI Dashboard tool, and (#5) By applying the methodology/tools in 8 case studies, integrated under one virtual, smart-city-like, European case study. The SCIs considered (in 8 European countries!) deal with energy, transportation, health, and water.

This approach will allow benchmarking the best-practice solutions and identifying the early warnings, improving resilience of SCIs against new threats and cascading and ripple effects. The benefits/savings to be achieved by the project will be assessed by the reinsurance company participant. The consortium involves seven leading end-users/industries in the area, seven leading research organizations, supported by academia and lead by a dedicated European organization. External world leading resilience experts will be included in the Advisory Board.


page iii

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Executive Summary

This report describes the stress-test and evaluation framework to be used by the case studies in tasks 5.3-5.11. It describes how to stress-test the Smart Critical Infrastructures (SCIs), and how to evaluate the results.

It introduces the resilience-based approach to stress-testing, and gives a step-by-step description of the stress-test methodology, which is developed in T3.3. The methodology is accompanied with support tools in the database, foremost the creation and use of dynamic checklists (DCLs). A DCL procedure is provided in the report.

Each of the nine case studies – ALPHA to INDIA – in tasks 5.3-5.11 have provided one or more specific stress-test scenarios, which will be carried out as an event (workshop, table top simulation, game, or drill), covering one or more of the most relevant type of threats for these smart critical infrastructures (SCIs).

Required functionalities and corresponding thresholds (minimum functionality) have been determined in this stress-test framework report – prior to performing the stress-tests.

Thus, after conducting the stress-tests, it can be evaluated whether the thresholds or limits have been exceeded or not, i.e. whether the SCIs have failed or passed the stress-tests. The evaluation is documented in the evaluation report, for which a format is provided in this report. This includes an evaluation/feedback on the stress-test methodology and the corresponding support tools.


page iv

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Table of Contents

1.1 About the report and the intended reader ....................................................... 11 1.2 Stress-test of critical infrastructures ................................................................. 12 1.3 Terms and definitions ......................................................................................... 14 1.4 Report structure ................................................................................................. 15

2.1 Method steps ...................................................................................................... 21 2.1.1 Step 1: Define the scenario .................................................................... 22 2.1.2 Step 2: Start creating the dynamic checklist (DCL) – DCL Step 1 ......... 23 2.1.3 Step 3: Select the SCI to be assessed – DCL Step 1 ............................... 24 2.1.4 Step 4: Select relevant threats for the selected SCI – DCL Step 1 ........ 25 2.1.5 Step 5: Specify the main stress-test criteria .......................................... 25 2.1.6 Step 6: Define the functionality elements – DCL Step 2 ....................... 26 2.1.7 Step 7: Define appropriate functionality indicators for each element

– DCL Step 3 ............................................................................................ 26 2.1.8 Step 8: Assign values for the functionality indicators – DCL Step 4 ..... 28 2.1.9 Step 9: Run the calculation and save the "Functionality Level

assessment" – DCL Step 4 ...................................................................... 30 2.1.10 Step 10: Compare the FL against the criteria and evaluate the

stress-test result ..................................................................................... 32

4.1 Stress-test staff and management .................................................................... 35 4.1.1 General Coordination Team ................................................................... 35 4.1.2 Local Coordination Unit .......................................................................... 35 4.1.3 Test Support Unit .................................................................................... 36 4.1.4 Guests ...................................................................................................... 36

4.2 Public relations ................................................................................................... 36 4.3 Evaluation and reporting .................................................................................... 37


page v

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.1 ALPHA (“Financial system”), City of Edinburgh, UK .......................................... 38 5.1.1 Basic description of Scenario ................................................................. 38 5.1.2 Functionality elements, Functionality indicators and limits

(thresholds) ............................................................................................. 38 5.1.3 Detailed description of stress-test scenarios ........................................ 39

5.2 BRAVO (“The smart city” - The future-oriented and sustainable community of Bahnstadt), Heidelberg, Germany ................................................................. 41 5.2.1 Basic description of Scenario ................................................................. 41 5.2.2 Functionality elements, Functionality indicators and limits

(thresholds) ............................................................................................. 41 5.2.3 Detailed description of stress-test scenario/s ....................................... 42

5.3 CHARLIE (“Health care system”), Austria .......................................................... 45 5.3.1 Basic description of Scenario ................................................................. 45 5.3.2 Functionality elements, Functionality indicators and limits


5.4 DELTA (“Airport”): Air transport infrastructure, Hungary ................................ 50 5.4.1 Basic description of scenario .................................................................. 50 5.4.2 Functionality elements, Functionality indicators and limits


5.5 ECHO (“Refinery”) - NIS refinery-industrial plant, Serbia ................................. 57 5.5.1 Basic description of scenario .................................................................. 57 5.5.2 Functionality elements, Functionality indicators and limits


5.6 FOXTROT (“Water”) - Drinking water supply in Sweden .................................. 59 5.6.1 Basic description of scenario .................................................................. 59 5.6.2 Functionality elements, Functionality indicators and limits


5.7 GOLF (“Transport”) – Flooding in the City of Cork, Ireland .............................. 61 5.7.1 Basic description of scenario .................................................................. 61 5.7.2 Functionality elements, Functionality indicators and limits


5.8 HOTEL (“Energy”) - Energy supply infrastructure in Helsinki, Finland ............. 63 5.8.1 Description of SCI and most relevant threats ....................................... 63 5.8.2 Critical functionality, functionality indicators and thresholds ............. 63 5.8.3 Detailed description of stress-test scenario .......................................... 63

5.10 INDIA (Integrated case) ...................................................................................... 65 5.10.1 Description of fictitious city (multiple SCIs)........................................... 65 5.10.2 Functionality elements, Functionality indicators and limits


DCL procedure .................................................................................................... 71


page vi

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Details of the DELTA exercise (example) ........................................................... 76

Response to internal review process ................................................................. 82

Resilience Joint Evaluation and Test Report (JET report) ................................. 83


page vii

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

List of Figures

Figure 1: Stress-testing SCIs using the resilience curve (based on Kröger (2014) [7]) ..... 12 Figure 2: Representation of the network supply and demand curves after a natural

disaster event (from STREST D5.4 [3], adapted from Sun et al. 2015 [21]). ..... 14 Figure 3: Structure to define the functionality of the infrastructure ................................ 16 Figure 4: Examples of Functionality elements and indicators ........................................... 17 Figure 5: Functionality level as a function of scenario time .............................................. 18 Figure 6: Level of functionality as stress-test limit ............................................................. 19 Figure 7: Time as stress-test limit ....................................................................................... 20 Figure 8: Loss of functionality surface as stress-test limit ................................................. 21 Figure 9: Creating dynamic checklist (DCL) in the database ............................................. 23 Figure 10: Basic information in DCL from Steps 2-4 (and Step 1) ....................................... 24 Figure 11: Defining the functionality elements (DCL Step 2) .............................................. 26 Figure 12: Defining the functionality indicators (DCL Step 3) ............................................. 27 Figure 13: Assigning values for the indicators (DCL Step 4) ................................................ 29 Figure 14: Run the calculation and obtain FL versus time curves (DCL Step 4).................. 31 Figure 15: Comparing the FL curve with the stress-test criteria ......................................... 32 Figure 16: Exercise Structure, ISO 22398, 2011 [5] ................................................................... 43 Figure 17: Map of Austria with the position of all healthcare provider. Each of the circles

represents a general practitioner (turquoise), medical specialist (orange) or hospital (dark blue). ............................................................................................. 47

Figure 18: Map of healthcare provider in Vienna for a given day on the data. The size of the circles indicates the number of patients treated by the given provider. ... 47

Figure 19: To assess the ability of the healthcare system to cope with a sudden and massive in-flow of patients we evaluate how efficient the system is able to distribute flows of patients based on naturally occurring ................................. 48

Figure 20: Based on this stress test, we assess the contributions of each individual healthcare provider to the decrease in its functionality level. These contributions are visualized here for each provider as different colors, ranging from green (low impact on functionality level) over blue (medium impact) to red (provider is critical for the functionality level of the health care system). ........................................................................................................ 49

Figure 21: Budapest airport terminal 1 division to airside and landside (other markings are representing shops, are from the original opera and not relevant for this deliverable) .................................................................................................... 51

Figure 22: Possible deployment of volunteers ..................................................................... 53 Figure 23: Evacuation of passengers and staff away from threat – out of terminal .......... 54 Figure 24: Evacuation of passengers and staff away from threat – into the terminal ....... 55 Figure 25: Evacuation of passengers and staff – towards the landside .............................. 56 Figure 26: Cascading effects among multiple SCIs in fictitious city .................................... 65 Figure 27: Comprehensive visualization of FL change ......................................................... 66 Figure 28 visualization of the event chain and affected SCIs.................................................... 67


page viii

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

List of Tables

Table 1: Stress-test method steps ..................................................................................... 21 Table 2: Definition of scenario (examples) ........................................................................ 22 Table 3: Critical infrastructure sectors or subsectors covered in SmartResilience ......... 24 Table 4: Typical threats ...................................................................................................... 25 Table 5: Stress-test cases in SmartResilience ................................................................... 33 Table 6: Evaluation and reporting – tasks and timing ...................................................... 37 Table 7: Smartness level of airports .................................................................................. 50


page ix

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

List of Acronyms

Acronym Definition A Area

AIA Applied Intelligence Analytics

BLEVE Boiling liquid expanding vapour explosion

BUD IATA code for Budapest Airport

BZN Bay Zoltan Nonprofit Ltd. for Applied Research

CCTV Closed Circuit Television (video surveillance)

CET Central European Time

CI Critical Infrastructure

CoE City of Edinburgh

CVE Common Vulnerabilities and Exposures

D Deliverable

DB Database

DCL Dynamic Check List

EOD Explosive Ordnance Disposal

EU European Union

EU-VRi European Virtual Institute for Integrated Risk Management

FhG-INT Fraunhofer Gesellschaft e.V. - Institute for Technological Trend Analysis

FE Functionality Elements

FI Functionality Indicators

FL Functionality Level

FP Framework Programme

HCP Health Care Provider

HELEN Helsingin Energia

HNP Hungarian National Police

HQ Headquarters

IBM International Business Machines

ID Identification

IED Improvised Explosive Device

INFRARISK Novel indicators for identifying critical INFRAstructure at RISK from natural hazards

ISO International Standardization Organization

IVL Swedish Environmental Research Institute

JD Juris Doctor

JET-report Joint Evaluation and Test Report

LPG Liquefied Petroleum Gas

MUW Medical University of Vienna

NIS Petroleum Industry of Serbia

PU Public

RI Resilience Indicators

RPP Responsible Project Partner


page x

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Acronym Definition R-Tech Steinbeis Advanced Risk Technologies GmbH

SCI Smart Critical Infrastructure

SINTEF The Foundation for Scientific and Industrial Research

SRH Heidelberg University of Applied Sciences

STREST Harmonized approach to stress tests for critical infrastructures against natural haz.

SWH Stadtwerke Heidelberg

t time

T Task

T1 Terminal 1 of BUD

VTT Technical Research Centre of Finland Ltd.

WP Work Package


page 11

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Introduction

1.1 About the report and the intended reader This report describes the stress-test and evaluation framework to be used by the case studies in tasks 5.3-5.11. It describes how to stress-test the Smart Critical Infrastructures (SCIs), and how to evaluate the results. Thus, the main intended reader are the case study partners.

The objective is to stress-test the SCIs selected in the SmartResilience project, not the methodology or tools (or software) in WP3; however, the stress-test methodology that is used to stress-test the SCIs is developed in T3.3 (prediction/modelling), and the stress-test is at the same time a testing of this methodology.

The main aim of WP3 is to develop an indicator based methodology and an integrated tool for assessing, predicting and monitoring resilience of SCIs. Thus, there are three pillars of methodology being developed:

1. Resilience assessment (in T3.2) 2. Functionality assessment – stress-testing (in T3.3) 3. Monitoring and optimization (in T3.4)

All relevant parts of the methodology and the related support tools are meant to be tested in WP5 by the case studies (T5.3-5.11); however, this report is only about the second pillar of methodology providing the stress-test method.

The resilience assessment methodology developed in T3.2 [16] and related supporting tools developed in T3.7, including dynamic checklists (DCLs), which are described in the resilience assessment guideline (D3.6) [17], are relevant objects for testing in WP5. Both the resilience assessment (T3.2) and the stress-testing (T3.3) can be performed as on-time assessments/tests, whereas monitoring (T3.4) will require repeated assessments/ measurements over a certain time period. Thus, extensive testing of monitoring and optimization may be challenging to obtain during the WP5 testing period, and the solution may be some simplified testing of this third pillar of the methodology.

However, as stated above, this report is only about the stress-test of the SCIs, using the stress-test methodology developed in T3.3. Here the focus is on the resilience curve itself, as illustrated in Figure 1.


page 12

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Figure 1: Stress-testing SCIs using the resilience curve (based on Kröger (2014) [7])

Depending on the development of the functionality level during a stress-test scenario and the thresholds/ limits set (both in functionality level as shown in Figure 1, but also in time/duration, or both) an SCI passes or fails a stress-test.

For the SmartResilience project, the main goal is not to evaluate the SCIs, but to harvest data emanating from the functionality level change during scripted events, pre-set scenarios and feed those into DCLs, testing the methodology with information acquired in controlled test environment and event flow (as a basic need for a scientific experiment). It is not crucial that all case studies are testing all parts of the methodologies and corresponding support tools. For those case studies that are performing in-depth stress-testing (see Chapter 3), other testing, such as resilience level assessment (T3.2) are optional.

1.2 Stress-test of critical infrastructures The stress-testing curve is not unique for SmartResilience, what is unique with the stress-testing methodology in SmartResilience is how the curve is obtained, using a methodology closely linked to the resilience assessment methodology, and a supporting tool for functionality assessment using dynamic checklists (DCLs). This is described in Chapter 2.

Two recent EU projects, INFRARISK1 and STREST2, carried out in FP7, have looked into stress-testing of critical infrastructures3. INFRARISK D6.1 [1] has made a review of definitions of stress-test and states that it is about testing of the system/structure under abnormal (exceptional) conditions. One of the referenced definitions, from Dictionary.com4, states that stress-testing is:

1 INFRARISK - Novel indicators for identifying critical INFRAstructure at RISK from Natural Hazards (2013-2016) 2 STREST - Harmonized approach to stress tests for critical infrastructures against natural hazards (2013-2016) 3 STREST also looked into and built further on previous EU FP7 projects (GEISER, MATRIX, NERA, REAKT, SHARE,

SYNER-G), and on-going projects from the FP7 2013 call (ASTARTE, INFRARISK, INTACT), Mignan, 2014 [10]. 4 http://dictionary.reference.com/browse/stress+test?s=t

http://dictionary.reference.com/browse/stress+test?s=t


page 13

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

A simulation test to determine how a given institution, system, etc., would perform under greater than usual stresses or pressures: the government's stress test for big banks; mandatory stress tests for nuclear plants; a website stress test that simulates peak traffic.

It is quite common to refer to stress-test of banks/finance and nuclear plants, and frequently reference is made to the Fukushima accident to highlight the need for stress-testing. INFRARISK D6.2 [23] refers to the Fukushima Daiichi plant as a missed opportunity for stress-testing: the levees were built to withstand typhoon related 5.7-meter wave heights, while the exceedance return period of 10 meter tsunamis was only a mere 36 years, as 14 tsunamis of at least 10 meters have been observed in the last 500 years on the Japanese west coast (Krauß, M., Berg, H.P., 2011 [6]). It can be argued that a stress-test exercise prior to the 2011 tsunami might have given the Fukushima Daiichi stakeholders a timely warning to implement improvement measures.

A main difference between INFRARISK and STREST is that the former is following a risk based approach, whereas STREST turns towards a resilience based approach. INFRARISK D6.2 [23] states that infrastructural systems will be modelled as fault tree systems, and that a stress test is a special instance of a risk assessment. In a stress test, one assumes that some extreme hazard event has actually occurred, whereas in a risk assessment the effects of all hazard scenarios are accounted for and weighted by their respective probabilities of occurrence.

STREST takes a different stand [3], stating that: Today, after decades of development of probabilistic risk assessment techniques, there are solid probabilistic risk assessment methods and tools that provide practical estimates of instantaneous CI performance (service) loss due to direct and indirect disaster-induced damage. However, the instantaneous loss by itself does not reveal how a community served by the CIs responds to a disaster. The time dimension represents a key aspect: the time-evolution of community needs and the ability of the CIs to fulfill these needs (e.g. water, gas, and electricity) is best represented and modelled using the concept of resilience rather than of risk.

Thus, STREST advocates a resilience-based stress test concept, which is very much in line with what is done in SmartResilience.

Regarding the stress-test limits or thresholds, e.g. illustrated in Figure 1, STREST has an interesting viewpoint. They state that for the identification of resilience-based acceptance criteria, understanding how the different stakeholders' needs depend on the functionality of the CIs represents the key issue. However, they also realize that taking into account the time evolution of the CIs and the community systems (i.e. the demands, that may not be static) during the recovery process, is not trivial.

Communities and CI systems are complex systems of systems. Most of the resilience quantification frameworks proposed in literature impose the point of view of the infrastructure owner, i.e. to recover the initial functionality of the system as fast as possible. However, CIs are built to deliver a service to a community, then the resilience assessment should also take into account the ability of a CI to supply the time-varying community demand for the services provided by the assessed CI (Mieler et al., 2015 [9]). The resilience becomes dependent not only on the performance and ability of the CI to recover, but also on the ability of the community to recovery the demand. This is illustrated in Figure 2 [3].


page 14

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Figure 2: Representation of the network supply and demand curves after a natural disaster event (from

STREST D5.4 [3], adapted from Sun et al. 2015 [21]).

Loss of resilience is not only a function of loss of CI functionality (supply), but also a function of community demand. I.e., it is not the area between the nominal CI functionality (typically the 100 % horizontal functionality line) and the loss of CI functionality (supply), but the area between the two curves. As stated by STREST, this is not trivial, and in SmartResilience, we have so far focused on the CI loss of functionality, i.e. only the blue curve in Figure 2.

This means that we see it more from the infrastructure owners point of view, than from the community point of view, which simplifies the stress-test methodology, although it is still not quite straight forward.

The SmartResilience stress-test methodology is described in Chapter 2.

1.3 Terms and definitions Definitions are based on a selection of the definitions in ISO 22300 [4] and ISO22398 [5], with necessary extensions to comply with the project vocabulary and test purposes. See also the Glossary http://www.smartresilience.eu-vri.eu/ (member area).

Drill: Activity which practices a particular skill often involves repeating the same thing several times.

Documentation Coordinator:

Person responsible for establishing and leading a small group of evaluators and/or observers for measuring and recording the selected indicators and elaborate them during the post-exercise activities.

Events: Occurrence happening at a determinable time and place, with or without the participation of human agents. Events describe situations arising from the time horizon, manifesting the general contents of the exercised scenario. In other words, the events provide a general overview of the scenario. The number of events depends on the performance objectives


page 15

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

and aims of the exercise. In case of most of the exercises described in this document, events will be represented by injects.

Exercise: Process to train for, assess, practice, and improve performance in an organization. In the current context, exercises will be used for: validating procedures, clarifying and identifying opportunities for improvement and controlled opportunity to practice use of resilience indicators developed in the project.

Exercise Coordinator:

Person responsible for planning, execution, and evaluation activities of an exercise, also responsible for the cooperation among internal and external entities. In the current context, the exercise coordinators are the respective task leaders of tasks 5.3-5.11.

Incident: Events with consequences that may have negative impact on the operational level or business relations of the affected critical infrastructures.

Inject: Scripted piece of information inserted into exercise designed to elicit a response or decision and facilitate the flow of the exercise.

Joint Evaluation and Test Report (JET report):

After-action report document which records, describes and analyses the exercise, drawing on debriefs and reports from evaluators, participants, and observers derives lessons from it.

Observer: Exercise participant who watches selected segments as they unfold while remaining separate from role player activities. In the current context, the observers will record data potentially relevant for resilience and/or functionality indicators as well as the flow of events through the phases.

Safety: Status or condition of people, property, information and operation being protected against or from intentional, unintentional human act or natural disaster.

Scenario: Pre-planned storyline that drives an exercise, the stimuli used to achieve exercise objectives.

Script: Story of the exercise as it develops which allows directing staff to understand how events will develop during exercise play as the various elements of the master events list are introduced.

Test: A test is a unique and particular type of exercise, which incorporates an expectation of a pass or fail element within the goal or objectives of the exercise being planned.

1.4 Report structure Chapter 1 provided a brief introduction to the report, including some previous research on stress-testing of critical infrastructures. Chapter 2 provides information about the stress-test methodology in SmartResilience. Chapter 3 gives an overview of the stress-test cases, and Chapter 4 gives some information on organizational arrangements for the stress-tests. Chapter 5 covers the description of each of the nine stress-test cases (SCIs and scenarios), including detailed outline of the stress-test scenarios, provided by the task leaders for each of the test case studies. The annexes include support material and additional details about the stress-test cases.


page 16

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Stress-test methodology in SmartResilience

The resilience curve, e.g. as shown in Figure 1, describes the development of functionality level versus time, when an event occurs. The functionality of a critical infrastructure is the role of the infrastructure, and it is a critical infrastructure if loss of functionality has a severe impact on society. It is not straight forward to decide exactly what the functionalities for a given SCI are, as this ranges from simply its existence and integrity (a dam protecting from flood) to complex facilities providing services with coverage area and network (healthcare system). In addition, main functions can be broken down in sub-functions, sub-sub functions, and so on, and finally into functionality elements that can be measured by (functionality) indicators.

For simplicity, we select what is considered the most relevant functions/functionalities and we only use two levels in addition to the overall functionality level: functionality elements and functionality indicators.

The method proposed to assess functionality, pertinently to the overall methodology in SmartResilience, proposes by do it by means of

• Functionality elements (FE) are “single functionalities” that contribute to the overall functionality of the CI (correspond to the issues in the main SmartResilience method for defining the resilience level) and

• Functionality indicators (FI) provide the values in order to measure the elements (in this particular case the “the indicators of functionally” or “the indicators of functionality performance”).

Functionality of the Infrastructure

Functionality Element (FE)

Functionality Indicator (FI)

Defined as a portfolio of single functionalities

Indicators provide the values in order to measure the functionality elements

Functionality elements are “single functionalities” that contribute to the

overall functionality of the CI

Figure 3: Structure to define the functionality of the infrastructure

NOTE: This portfolio of functionality elements and functionality indicators Figure 3, shall normally NOT change over the resilience cycle. i.e. once defined to be representative, it will be used until the end of the cycle.


page 17

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Functionality Level (FL) of

Infrastructure

GLOBAL: international/

interconnectedness

GOVERNANCE: Human related &

governance

SOCIAL/SOCIETAL performance

HSE:Health, Safety and

Environment performance

SECURITY:Security system

performance

ECONOMY: Economic

performance

CORE: material production / output

performance

e.g.Nominal daily production (t/day)

e.g.Direct greenhouse gas emissions (t/year)

e.g.Revenues (€/day)

e.g.Are data centres protected by shielding and filtering? (yes/no)

e.g.Did the incident affect other similar/related plants? (yes/no)

e.g.Stakeholders engagement maintained (yes/no)

e.g.Implementation of resilience policy (yes/no)

Indicator 1

Indicator 2

Indicator 3

IT: Information

technology & communication

Indicator 1

Indicator 2

Indicator 3

Indicator 1

Indicator 2Indicator 3

Indicator 1


Indicator 1


Indicator 1


Indicator 1


Indicator 1


e.g.No. of firewalls in the cyber security system? (no. / company)

LEVEL 2 LEVEL 4 LEVEL 5

Figure 4: Examples of Functionality elements and indicators


page 18

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

A functionality assessment can be performed both during normal operation and during an event; however, a stress-test is performed by imposing an (imagined) severe threat/event on a SCI and assess whether the SCI is resilient enough to be able to continue functioning within prescribed limits or not. To do this, we need to measure the curve itself or some parameters of the curve. In SmartResilience, we aim at measuring the functionality level at six different points in time, which then describes the resilience curve as illustrated in Figure 5.

Figure 5: Functionality level as a function of scenario time

The points in time are as follows:

• t0: time before the event (nominal level of functionality) • t1: time at which the event occurs • t2: time at which the infrastructure lost its full functionality or part of its functionality • t3: time at which the infrastructure starts to recover • t4: time at which the infrastructure reaches the initial functionality level (nominal level) • t5: time at which the infrastructure reaches its highest level through learning and adaptation

In addition, we need to decide on the thresholds/limits representing acceptable / non-acceptable loss of functionality, given a threat/event. The stress-test criteria can be related to:

• Level of functionality (overall, functionality elements and/or functionality indicators) • Time (to absorb, recover) • Surface (accumulated "loss of functionality")


page 19

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Level of functionality ("vertical loss")

This is illustrated in Figure 6.

Figure 6: Level of functionality as stress-test limit

The stress-test limits can be set on overall functionality level, at single functionality element(s), and/or at single functionality indicator(s). The level at the bottom of the curve is sometimes referred to as "robustness", which can be set as a stress-test limit.


page 20

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Time ("horizontal loss")


The time limits can be e.g. maximum time to absorb the event, maximum time to partially recover after the event, or maximum time to fully recover after the event. The last time interval, i.e. time between the event occurs and the SCI is fully recovered, is sometimes referred to as "rapidity". This can typically be used as stress-test limit. Another typical measure is time to 90 % functionality is achieved, which also can be used as stress-test limit.

Figure 7: Time as stress-test limit


page 21

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Surface ("loss of functionality area")


Figure 8: Loss of functionality surface as stress-test limit

The maximum allowed accumulated loss of functionality is illustrated as area A2. The actual (or assumed/ assessed) loss of functionality area can be smaller (A1) or larger (A3) than the limit, leading to either success or failure of the stress-test.

2.1 Method steps The 10 method steps for performing stress-tests are shown in Table 1. They are defined as "similar" steps to the resilience assessment method for determining resilience levels described in D3.2 [16]. The method steps are integrated with the supporting tool, which is the dynamic checklist (DCL) tool. This tool is used for some of the method steps, in four DCL steps as shown in the third column.

Table 1: Stress-test method steps

Method step no. DCL step no. Description

Scenario 1 - Define the scenario

Dynamic checklist

2 Step 1: Basic info

Start creating the dynamic checklist (DCL)

3 Select the SCI to be assessed


page 22

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

4 Select relevant threats for the selected SCI

5 - Specify the main stress-test criteria

6 Step 2: Elements Define the functionality elements

7 Step 3: Indicators Define the appropriate functionality indicators for each element

8 Step 4: Fill out

Assign values for the functionality indicators

9 Run the calculation and save the "Functionality Level assessment"

Results 10 - Compare the FL against the criteria and evaluate the stress-test result

2.1.1 Step 1: Define the scenario

Scenario is a general term that can be used in different ways. In SmartResilience, we use the term scenario for a specific selection of critical infrastructures and threats for a given area/city, i.e. the selected geographical area, sector, critical infrastructures and threats. This is necessary and sufficient for baseline resilience assessment.

However, for functionality assessment, and in particular for stress-testing, it is necessary to define a detailed stress-test scenario. It is not sufficient to describe what (i.e. the threat(s)), it is also necessary to describe where, when, how severe, etc. This is a more traditional way of defining "scenario". It can also be a combination of different threats, e.g. a combination of a cyber-attack and a terror attack. This detailed stress-test scenario description is not required for the DCL, but it is necessary when assigning values for the functionality indicators in Step 8. The detailed stress-test scenario description can be considered as an extension to Step 1.

The general scenario definition can be assisted by the following sub-steps:

a. Scenario name b. Scenario description c. Type of SCIs (sector) d. SCIs involved (sub-sector, facility) e. Type of threats

Examples are provided in Table 2.

Table 2: Definition of scenario (examples)

Sub-steps Example 1 (DELTA) Example 2 (ECHO)

1a Scenario name Smart airport, Hungary Smart industrial zone, City of Pancevo, Serbia

1b Scenario description The Budapest Liszt Ferenc International Airport is an air transportation system embedded to a Smart City and operating as Smart Critical Infrastructure. It has to be assessed in case of blocked traffic and exceeding capacity.

The smart industrial zone in the city of Pancevo consists of different infrastructures such as NIS oil refinery, Azotara, HIP, etc., i.e. multiple SCIs.

1c Type of SCIs (sector) Transportation Production and energy

1d SCIs involved (sub-sector)

Air transportation - airport Production and storage of chemical substances / oil refinery

1e Type of threats Cyber-attack, terror attack, explosion Cyber-attack, terror attack, explosion


page 23

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

The detailed stress-test scenario descriptions are included in Chapter 5 for each case study.

2.1.2 Step 2: Start creating the dynamic checklist (DCL) – DCL Step 1 The dynamic checklist is created in the integrated tool / database (in the SmartResilience member area), as illustrated in Figure 9. A procedure for producing a DCL is included in Annex 1.

Figure 9: Creating dynamic checklist (DCL) in the database

The creation of a DCL starts by providing basic information (DCL Step 1: Basic info) as illustrated in Figure 9, which includes the stress-test method steps 2-4 (as shown in Table 1). Information from Step 1 (Table 2) is also included, as illustrated in Figure 10.


page 24

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Figure 10: Basic information in DCL from Steps 2-4 (and Step 1)

2.1.3 Step 3: Select the SCI to be assessed – DCL Step 1

This is the SCI (or SCIs) that will be stress-tested. This is the same as provided in Step 1c, when defining the scenario. In SmartResilience, we cover the SCIs shown in Table 3.

Table 3: Critical infrastructure sectors or subsectors covered in SmartResilience

Sector Subsector Sub-subsector

Financial

Payment services/payment structures (private)

Government financial assignment

Energy Transmission of electricity

Distribution of electricity

Underground coal storage *

Healthcare Medical and hospital care

Transportation Air traffic (air transportation) Airport

Production Production and storage of chemical substances **

Step 2

Step 3 / Step 1c

Step 4 / Step 1e

Step 1a

Step 1b

Step 2

Step 2

Step 2 / Step 1d


page 25

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Sector Subsector Sub-subsector

Water Provision of drinking water

Control of water quality

ICT Information systems and networks protection

Internet

* Special sub-sector added for the HOTEL case

** An oil refinery is also considered as a sub-sector within Energy

An overview of all types of critical infrastructure sectors and subsectors can be found in the D3.6 resilience assessment guideline [17].

Table 3 shows that no single sector is covered entirely; only selected subsectors (and in one instance a sub-subsector) are covered in the SmartResilience project. These are shaded green in the table.

2.1.4 Step 4: Select relevant threats for the selected SCI – DCL Step 1

Select the most relevant types of threats for each SCI. This is the same as provided in Step 1e. The specific stress-test scenario(s) should be related to one or more of these types of threats. Typical threats to consider are included in Table 4. Examples were provided in Table 2.

Table 4: Typical threats

Type of threat Specific threat

Terrorist attack Bombing attempt

Cyber attack FireDos attack

Natural threats (incl. extreme weather)

General

Urban floods

Other (specify)

New technology related threats Explosion

Other (specify)

Other Interruption in critical supply

Other (specify)

2.1.5 Step 5: Specify the main stress-test criteria

This was described and illustrated in Figure 6 – Figure 8. First, we need to determine the functionalities that are depicting the resilience curve(s), i.e. the y-axis. Second, we need to determine what constitutes 100 % functionality level, and third, we need to decide on which stress-test criteria to use. It could be a certain minimum level of functionality (i.e. the lowest point of the resilience curve should be above this FLmin), it could be the time from the event occur until 90 % of the functionality is restored, or some combination of various criteria.

Examples related to the DELTA case (airport) could be:

• The overall level of functionality should not be below 10 % of the normal level of functionality (runway intact, aircrafts can perform emergency landing)

• The evacuation time should not be more than 20 minutes (from announcement to sealing) • The maximum allowed "loss of functionality" should not be more than 77 %


page 26

Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

2.1.6 Step 6: Define the functionality elements – DCL Step 2 In SmartResilience, the functionality of each SCI is defined as a portfolio of functionality elements (FEs), which may include some core single functionalities, such as production and/or some supporting elements such as safety and/or environmental performance or similar. Defining the FEs is Step 2 in the DCL tool, as illustrated in Figure 11.

Figure 11: Defining the functionality elements (DCL Step 2)

Relevant functionality elements are searched for, selected and included in the DCL. In Figure 11, it is highlighted that this is done by marking the FE that is selected, and then drag and drop it to the name of the SCI (top level); in this example called "NIS refinery functionality assessment".

2.1.7 Step 7: Define appropriate functionality indicators for each element – DCL Step 3

For each functionality element, we need to define the indicators that will be used to measure the elements, i.e. the functionality indicators (FIs). This could be e.g. tons of oil produced, number of passengers processed, amount of electricity produced, etc. Defining the appropriate FIs is Step 3 in the DCL tool, as illustrated in Figure 12.

Relevant functionality indicators are searched for, selected and included in the DCL. In Figure 12, it is highlighted that this is done by marking the FI that is selected, and the drag and drop it to the functionality element it is measuring. In the example shown, the FI selected belongs to the FE "Production: Material production / output performance".


page 27

w

ork

Figure 12: Defining the functionality indicators (DCL Step 3)


page 28 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

2.1.8 Step 8: Assign values for the functionality indicators – DCL Step 4 These are the measured, assumed or assessed values of the functionality indicators during the course of the event from it occurs until recovery, including the values obtained after learning and adaptation. The values are measured, assumed or assessed at six different points in time, as was shown in Figure 13.

This step constitutes the first part of the fourth and final step in the DCL tool termed "Fill out", referring to filling out the six values for each indicator. This is illustrated in Figure 13.

It is a natural phenomena for facilities, that after an adverse effect, damaged or destroyed equipments are replaced by new parts, which, due to the advance of technology and their new state, may provide better output than their predecessors. The same applies for methodologies and Standard Operation Procedures, they will be revised during the adapt and learn phase, possibly resulting in optimization or even innovation leading to higher functional level than before the adverse effect (disaster). This is same as if you buy a new car or washing machine, it is usually performing better than the old, worn one. To display this, we indicated higher functional levels at t5 that was at t1 actually.


page 29

Figure 13: Assigning values for the indicators (DCL Step 4)


page 30

2.1.9 Step 9: Run the calculation and save the "Functionality Level assessment" – DCL Step 4 The final step in the DCL tool is to run the calculation and obtain the functionality level versus scenario time curves for those "levels" (overall, element and indicator) where stress-test criteria are assigned. This last part of the final step (Step 4) in the DCL tool is illustrated in Figure 14.


page 31

Figure 14: Run the calculation and obtain FL versus time curves (DCL Step 4)


page 32 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

This is the result of the functionality level assessment, given a certain stress, that will be compared with the criteria (in Step 10) to determine whether the SCI has passed or failed the stress-test.

2.1.10 Step 10: Compare the FL against the criteria and evaluate the stress-test result

The functionality level versus time curve(s) obtained in the previous step (Step 9) are compared with the stress-test criteria in order to evaluate whether the SCI has passed or failed the stress-test.

Figure 15 illustrates four alternative FL versus time curves for one specific functionality (whether it is on overall, element, or indicator level). The stress-test criterion is in this case only related to the functionality level (not to time or surface/area).

Figure 15: Comparing the FL curve with the stress-test criteria

Only one alternative, the purple one ("FL: as before") is at all times above the stress-test limit, which means that only for this alternative the stress-test is passed. All other alternatives are passing the limit; thus, they have failed the stress-test. This is even the case for the orange alternative ("FL: better than before"), since the only stress-test criterion was the minimum functionality level. This shows that it is not necessarily straightforward to determine sensible stress-test criteria.

For the case illustrated in Figure 15, using only functionality level as stress-test criteria, the exact points in time of t0 to t5 are not important, as time is not part of the criteria. However, if time is used (as shown in Figure 7) or the surface for "loss of functionality" is used (as shown in Figure 8), then it is necessary to assign/log the values also for the times {t0, t1, t2, t3, t4, t5}, in addition to the corresponding FL values.


page 33 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Overview of stress-test cases

An overview of the stress-test cases is provided in Table 5, including exercise method, level and partner responsible for each case.

Table 5: Stress-test cases in SmartResilience

SCENARIO NAME SHORT DESCRIPTION EXERCISE METHOD LEVEL PARTNER

ALPHA Financial systems Workshop Brief BZN

BRAVO Community (loss of electricity) Tabletop Brief SWH

CHARLIE Healthcare (national level) Tabletop Brief MUW

DELTA Transportation (airport) Drill In-depth HNP

ECHO Industry (production site) Drill Brief NIS

FOXTROT Drinking water (microbial cont.) Workshop Brief IVL

GOLF Flood (pluvial flood) Workshop Brief AIA

HOTEL Storage (underground coal) Tabletop Brief VTT

INDIA Integrated scenario Game In-depth BZN

Any type of exercise is conducted using one or a combination of several methods. The current tests are using four types of exercise methods for the case studies. The methods are set in two categories: brief and in-depth.

Workshop (brief):

Workshops are designed to orient participants to new or updated plans, policies, or procedures. They are unconstrained by real-time simulation of events and are facilitated by an experienced presenter. Participant interaction is required, and the focus is on achieving a result. In the current context, it means invited experts of a critical infrastructure will analyze a given scenario in view of selected issues and indicators, providing sample data for the indicators where possible.

Tabletop simulation (brief):

A tabletop exercise will include key personnel discussing simulated scenarios that involve disruptive event in an informal setting [around a table]. Tabletop exercises can be a tool to build competence and support for a revised plan or procedure; or, review plans, policies, and procedures or to assess the systems needed to respond to undesired situations. Participants are expected to discuss the issues that result from the simulated events and develop decisions through paced problem solving. Tabletop exercises can be timed with expected rapid decision making or untimed allowing for in depth discussion and development of solutions. Usually, untimed tabletop exercises will be used in this project.


page 34 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Game (in-depth):

A game is a simulation of operations that often involves two or more teams, usually in a competitive environment, using rules, data, and procedures designed to depict an actual or assumed real-life situation.

Drill (in-depth):

A drill is a coordinated, supervised activity usually employed to test a single specific operation or function in a single entity or multi-organization team.

Note: As described in Section 1.1, for those case studies that are performing in-depth stress-testing, other testing such as resilience level assessment are optional.


page 35 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Organizational arrangements for the stress-tests

4.1 Stress-test staff and management To coordinate, organize, lead and perform the tests, a three-level exercise team structure is established temporary within the consortium, for all exercises with the number of team members dependent upon the type and method of exercise and available resources.

4.1.1 General Coordination Team

Tests will be performed in a coordinated manner, based on the time plan and basic outline decided at the Budapest Workshop on April 24-26, 2017. The coordination is the responsibility of the General Coordination Team consisting of:

• Work Package Leader of WP5 – Lt. Col. István Macsári (HNP) • Task leaders or deputy task leaders of tasks 5.3-5.11:

o ALPHA: Mary-Ellen Lang (CoE) o BRAVO: Sebastein Warketin(SWH) o CHARLIE: Peter Klimek (MUW) o DELTA: Lt. Col. Gábor Csapó, JD (HNP) o ECHO: Dmitrij Bezrukov (NIS) o FOXTROT: Johan Sanne (IVL) o GOLF: Thomas Knape (AIA) o HOTEL: Raija Koivisto (VTT) o INDIA: Székely Zoltán, JD (BZN)

The main tasks and responsibilities of the General Coordination Team are:

• elaborate general test protocol • to harmonize detailed test protocol of each test with other tests and the general test plan • review logistical needs of each test to ensure cost-effective logistics • determine venue of each test, ensuring that tests will not run in parallel with each other • ensure compliance with guidelines on social security exercises (ISO 22398 [5]) • provide results for task leader of T5.12

4.1.2 Local Coordination Unit

Local Coordination Units are group of experts contributing in organizing and execution of the tests under the leadership of the respective task leaders. It consists of:

• an exercise coordinator (the task leader); • a documentation coordinator; • controllers (organizers); • evaluators (observers); • exercise safety officer (for real-life exercises).


page 36 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

The main tasks of the Local Coordination Units are:

• elaborate detailed test protocol; • organize and lead the test; • allocate necessary human resources (test support unit); • collect logistical needs for the given test; • documentation of the test (including evaluation); • ensure data protection (e.g. informed consent of participants); • ensure compliance with guidelines on social security exercises (ISO 22398 [5]) for the given type of

exercise.

4.1.3 Test Support Unit

Each test has its own Test Support Unit, which is performing the actual exercise of the tests, under the leadership of the Local Coordination Unit. Depending of the type of the exercise it can consist of different number of people with various competences, for example a tabletop needs 3-5 experts, while a drill can require 1000 persons or more. It includes in particular:

• supporting actors (role-players); • observers; • reviewers; • technical personnel; • security staff (where needed at real-life exercises); • medical staff (where needed at large scale real-life exercises).

4.1.4 Guests

Guests are persons representing decision-makers, stakeholders, media and other parties invited by the Local Coordination Team. They visit the exercise for only a brief time, largely for internal or external PR purposes, and do not take part in the debrief.

4.2 Public relations In developing the exercises and testing communication strategy and setting performance objectives, the case study leaders should identify internal and external interested parties who are impacted by and/or who have expressed an interest in its activities, products, and services. It should also identify other potential interested parties with whom it wishes to communicate to achieve the overall performance objectives of its exercises and testing communication strategy.

Some examples of interested parties that could be considered by an organization include first responders and emergency management personnel, suppliers, public authorities, politicians and opinion leaders, schools, academics and researchers, professionals involved in environmental, health, and safety issues, media and other organizations.

It is not uncommon to identify conflicting interests among different interested parties. As a result, the exercises and testing activities should address and respond to different and often conflicting demands from interested parties, in particular those that are the most influential and who may negatively impact the outcomes of an exercises and testing activity.

When undertaking an exercises and testing activity, the organization should seek to comprehend the expectations and perceptions of interested parties with respect required competencies to perform the organizational mission. Direct dialogue between interested parties and the organization may generate the feedback required. If the organization is seeking input from interested parties, it should explain why it is seeking information, and what it plans to do with the information obtained.

The WP and Task leadership shall contact media organizations to reach the goals of the project and to let publish the exercise itself and the potential outcomes of the case studies.


page 37 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

4.3 Evaluation and reporting The task and time plan for the evaluation and reporting are described in Table 6. The report template is included in Annex 4.

Table 6: Evaluation and reporting – tasks and timing

PHASE NO. TASK TIME RESPONSIBLE

1 Post exercise de-briefing Shortly after the official closing of the exercises

Exercise coordinator

2 Evaluation report of the exercise (delivery)

Within 1 week Local Coordination Team

3 Data analysis Within 2 weeks General Coordination Team

4 Input for JET-report Within 1 month General Coordination Team


page 38 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Description of stress-test SCIs and scenarios

For each case study, the following information are provided:

1. Description of SCI and most relevant threats 2. Critical functionality, functionality indicators and thresholds 3. Detailed description of stress-test scenario

The description of each SCI and the most relevant threats are based on e.g. the D1.3 report [14] and the D5.1 report [18].

5.1 ALPHA (“Financial system”), City of Edinburgh, UK

5.1.1 Basic description of Scenario

The City of Edinburgh is Scotland’s capital city and has the second largest financial sector in the United Kingdom and the fourth largest market in Europe by equity assets. Next to London, Edinburgh is the biggest, financial technical (fintech) services hub in the UK, including banking, insurance and pensions, fund management and asset servicing. It generates around £8billion for the home economy per annum.

The city has several business areas servicing the financial sector.

The technology sector has expanded within the city, due to growing expertise within the city’s universities, and Edinburgh there are an estimated 17,136 people working within digital companies.

The financial services industry is using existing networks and new technologies to enhance their product offerings, service delivery and increase profits. As new technologies are adopted the interactions among related technologies are leading to ever-increasing complexity.

In this context, cyber-attack (both logical and physical) represents a serious threat, not only to disrupt financial services, but also crucially, to its dependent organizations – this includes all other critical infrastructure.

For this reason, cyber-attack has been selected as the most relevant threat to this sector for the purposes of this case study.

The principal cyber-attack threat may be a software related incident. Physical attacks may involve keyboard hacking, electronic pulses or malicious interference, e.g. changing time signatures on servers, or a combination of methods.

5.1.2 Functionality elements, Functionality indicators and limits (thresholds) The ALPHA case study is concerned with the resilience of the financial sector, focused on a cyber-attack scenario in or affecting Edinburgh

Due to the inherent dependencies of other organizations on the financial services sector, such a disruption would likely also negatively impact on other critical infrastructure, including healthcare, transport, utilities, etc.

This case study will use a cyber-attack scenario to determine the direct effects on financial services, as well as cascading effects, with particular reference to selected functionality element and indicators.

Examples of functionality elements that may be selected are:

1. Failure of key banking engines (e.g. core banking) 2. Ability to meet liabilities to pre-defined levels


page 39 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

3. Failure of systems supporting Straight Through Processing (STP) 4. Loss of income

Examples of functionality indicators that may be selected are:

1. Failure of key banking engines (eg core banking) 1.1. Number of core banking transactions – deposits 1.2. Number of core banking transactions – payments

2. Ability to meet liabilities 2.1. Ratio of assets (<) to liabilities

3. Failure of systems supporting Straight Through Processing (STP) 3.1. Number of core banking transactions – deposits 3.2. Number of core banking transactions – payments

4. Loss of income 4.1. Assessed profit margins per hour 4.2. Total income from all activities

A final set of functionality elements and functionality indicators will be selected and corresponding threshold values established prior to the workshop.

5.1.3 Detailed description of stress-test scenarios The City of Edinburgh is the capital city of Scotland. As at mid-2017, the estimated population of Edinburgh is 508,675 people.

Employing some 100,000 people (estimated 2016) and that figure again in support services, Edinburgh is the largest financial sector (by both gross value added and employment) in the United Kingdom, after London, and it is the fourth largest market in Europe by equity assets.

Outside of London, Edinburgh is the biggest, financial technical (fintech) services hub in the UK, with strong representation from banking, insurance and pensions, fund management and asset servicing. It is one of the country’s biggest sectors, generating around £8 billion for the home economy each year.

Included in Edinburgh’s financial sector are: the Bank of Scotland founded in 1695 and now part of the Lloyds Banking Group, with their Scottish headquarters in Edinburgh; the Royal Bank of Scotland (RBS) with its global headquarters in Edinburgh and including National Westminster Bank, Ulster Bank, Direct Line and Coutts (some of the operations that are part of the RBS group); Tesco Bank; and Virgin Money, both of which have headquarters in Edinburgh.

In insurance terms, Edinburgh companies such as Standard Life and Scottish Widows form a large part of the European insurance sector as well as being major employers in the city. Scottish Widows manages £145.79 billion worth of funds at June 2013 with a workforce of around 3,500.

Immediately to the west of the city centre, the Exchange business district, houses major employers such as Scottish Widows, Standard Life, the Clydesdale Bank, and Baillie Gifford.

Edinburgh Park is one of the largest business parks in the UK near Edinburgh Airport. Close to Edinburgh Park is the Royal Bank of Scotland global headquarters. HSBC, Royal Bank, Diageo, J. P. Morgan, Telewest, BT, Fujitsu and Lloyds Banking Group. Around 20,000 people work in the western outskirts of the city.

Edinburgh has an estimated 17,136 people working within digital companies as the technology sector has expanded on the back of the expertise within the city’s universities. The city has seen a growth in the number of software companies in the city over the last 10 years and there are now approximately 100. One of the main reasons behind the growth in the number of software companies is the highly skilled graduates from the University of Edinburgh’s world-leading School of Informatics. This area is also home to two tech incubators, Codebase and Techcube.

Stress test scenario case 1

The UK National Cyber Security Centre and other agencies internationally, are made aware that an email has been circulated to a number of high profile financial and technology sector companies, including a number headquartered in Edinburgh. The email is reportedly from an emergent, anti-capitalist, militant activist group (Helix5).

https://en.wikipedia.org/wiki/Bank_of_Scotland

https://en.wikipedia.org/wiki/Lloyds_Banking_Group

https://en.wikipedia.org/wiki/Lloyds_Banking_Group

https://en.wikipedia.org/wiki/Royal_Bank_of_Scotland

https://en.wikipedia.org/wiki/Ulster_Bank

https://en.wikipedia.org/wiki/Direct_Line

https://en.wikipedia.org/wiki/Coutts

https://en.wikipedia.org/wiki/Tesco_Bank

https://en.wikipedia.org/wiki/Virgin_Money_UK

https://en.wikipedia.org/wiki/Standard_Life

https://en.wikipedia.org/wiki/Scottish_Widows

https://en.wikipedia.org/wiki/Scottish_Widows

https://en.wikipedia.org/wiki/Standard_Life

https://en.wikipedia.org/wiki/Clydesdale_Bank

https://en.wikipedia.org/wiki/Baillie_Gifford

https://en.wikipedia.org/wiki/Edinburgh_Park

https://en.wikipedia.org/wiki/Edinburgh_Airport

https://en.wikipedia.org/wiki/HSBC

https://en.wikipedia.org/wiki/Diageo

https://en.wikipedia.org/wiki/J._P._Morgan

https://en.wikipedia.org/wiki/Telewest

https://en.wikipedia.org/wiki/BT_Group

https://en.wikipedia.org/wiki/Fujitsu

http://www.ed.ac.uk/informatics

http://www.thisiscodebase.com/

http://www.techcu.be/


page 40 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

The Activists are threatening to unleash a major attack on the various companies’ electronic systems utilizing an intentional electromagnetic interference (IEMI) capability unless the Group’s demands are met.

The Activist’s demands relate mainly to the adoption of ethical business practices, a wealth distribution programme and the establishment of an information exchange programme with less developed markets.

Through their own industry networks, the threatened financial and technology sector companies are ensuring that all subsidiary, partner and competitor companies are made aware of the threat.

Stress test scenario case 2

The media is reporting widely that a cyber-attack is currently taking place affecting a range of computer data centres and an estimated 250,000 individual computers across the UK.

Banks, financial service and technology companies appear to be the main targets of the attack but industry, health and government services are also being affected.

The attack is being spread on systems with inadequate security by various methods including phishing emails.

The form of attack is described as being similar to previous ransomware “WannaCry” incidents but more technically sophisticated. To date no demand for any payment has been made and the purpose behind the attack is not clear.

A group calling itself “GhostWatch” is claiming to be responsible for the incident but this information has not been verified. There is no known record of this organization.

Experts from the UK National Cyber Security Centre have warned the outbreak could continue to infect more systems.

Specialist investigators, including the UK National Crime Agency (NCA), are working with international counterparts to identify those responsible for the incident.


page 41 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.2 BRAVO (“The smart city” - The future-oriented and sustainable community of Bahnstadt), Heidelberg, Germany


Bahnstadt in Heidelberg is one of Germany’s largest urban development projects. It is designed to be Heidelberg’s first smart neighborhood. Bahnstadt is located in the south-western part of Heidelberg’s city center and shares a border with the main station. The energy concept consists of passive house standards as a universal construction method, district heating supply to be covered in the medium term by renewable energies, and intelligent control of power consumption using smart metering. Bahnstadt being the first smart neighborhood is dependent on the critical infrastructure, Stadtwerke Heidelberg (SWH) [20].

SWH identified three main hazards as potential threats:

1. Cyber-attack: Heidelberg, being a smart city, uses smart technologies such as smart grids to harness the potential of the new technologies. In this process, it is imperative to employ ICT resources and alternatively, making it vulnerable, too. Attacks through criminals hacking into the SWH data system (servers etc.) are increasing with the increased interconnectivity and globalization. These are driving greater frequency and severity of cyber incidents. The focus of SWH is mainly to prevent these attacks. In order deal with the challenges related to this threat, several physical measures are taken such as installation of security systems, firewalls for IT systems, security surveillance of fire-walls which have resisted well so far. This process requires, mobilizing the organization to take the resilience oriented measures, information sharing with the key stakeholders.

2. Terrorist attack: Heidelberg is a tourist attraction. With the increased terrorist activities in Europe, SWH foresees this as a prime threat as in the recent past such destinations became target of terrorist attacks. Such an attack could result in large loss of infrastructure. Hence, multi-fold efforts are undertaken by SWH to deal with the challenges in relation to this threat. The focus for SWH to address the challenges of ensuring that the physical infrastructure is well equipped to deal with any physical terrorist attack on the infrastructure or any cascading effects related to the disruptions in this scenario.

3. Flash flood: Heidelberg is a smart city in south-west Germany that lies on the River Neckar in a steep valley in the Odenwald. When the torrential rain occurs, the main challenge which city must deal with is how to proceed with downfall without damage for infrastructure or systems crucial to functioning of the city.

Furthermore, with the increasing use of smart technologies in the infrastructures such as smart grids, interconnected systems enabled with internet, efforts to globalize the economy, increased automation, inconsistent adoption of the smart technologies the smart critical infrastructures such as SWH also faces challenges related to increasing vulnerability to the new and emerging risks [21].

5.2.2 Functionality elements, Functionality indicators and limits (thresholds)

The stress-test scenarios imposed on the SCI will lead to degrees of loss of critical functionality measured by selected functionality indicators. The thresholds that are established reflect the acceptable levels of loss of critical functionality, given a stress. If the thresholds are exceeded, then the SCI has failed the stress-test.

Examples of functionality elements that may be considered selected are:

1. Cyber Security System Strength

2. Data Loss

3. Business Interruption

Examples of functionality indicators that may be considered selected are:

1. How many cyber-attacks have occurred in the past week? 2. How many cyber-attacks have been averted in the past week? 3. How many locations is the operations data being held?


page 42 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

4. How many unplanned grid shutdowns have occurred in the past year?

Examples of thresholds may be (examples not directly linked to the criticality elements and indicators):

1. The overall level of functionality should not be below 80 % of the nominal functionality level

2. The total production loss should not be more than 72 hours

3. The maximum allowed "loss of functionality" should not be more than 50 %

4. Percentage of households supplied with electricity falls below 98% of normal functionality.

A final set of functionality elements and functionality indicators will be selected and corresponding threshold values established prior to the exercise.

5.2.3 Detailed description of stress-test scenario/s

The Stadtwerke Heidelberg GmbH is divided into nine different private companies. The companies cover the regions of Heidelberg and Neckargemünd. The nine companies cover the main activities of providing electricity, gas, water, heating, public pools, and the mountain tram. The companies under the Stadtwerke Heidelberg GmbH umbrella are:

• Stadtwerke Heidelberg Energie (Energy) • Stadtwerke Heidelberg Netze (Network) • Stadtwerke Heidelberg Technische Dienste (Technical Services) • Stadtwerke Heidelberg Bäder (Pools) • Stadtwerke Heidelberg Umwelt (Environment) • Stadtwerke Heidelberg Garagen (Garages) • Heidelberger Straβen und Bergbahn (Streets & Mountain Tram) • Stadtwerke Neckargemünd • Stromnetz Neckargemünd (Electricity Netowrk)

The umbrella company Stadtwerke Heidelberg GmbH employs 214 people who generated 30 million euros in revenue in 2016. Stadtwerke Heidelberg Energie employs 31 employees that generate 190 million euros in revenue in 2016. Stadtwerke Heidelberg Technische Dienste employs 87 people who generate 3 million euros in revenue in 2016.The Stadtwerke Heidelberg Bäder employs 27 people while generating a revenue of 1.8 million euros in revenue. Stadtwerke Heidelberg Environment employs 25 people with a revenue generation of 31.8 million euros. Lastly, the Stadtwerke Heidelberg Garagen employs 2 people with total revenue in 2016 of 3.1 million revenue. In Heidelberg, the Stadtwerke is one of the largest employers with over 1,000 employees and 270.1 million euros in sales for 2016 [19].

The stress-test scenario will be executed by a table top exercise (TTX). This method was chosen as being both low risk and an effective exercise for problem solving. The planning of the TTX was done by following the guidelines given by the ISO/DIN 22398 [5]. As stated in the ISO 22398 the structure of the exercise needs to follow a top-down approach [5]. To start with is the objectives of the test, the scenario background, followed by the main events, incidents, and finally the injects. Figure 16 provides a depiction of the exercise structure [5].


page 43 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Figure 16: Exercise Structure, ISO 22398, 2011 [5]

In the end three individual scenarios were created and will be evaluated in the stress-test. The first is the scenario of a Flash flood. The city of Heidelberg is split in the middle by the Neckar river. With this beautiful river, the potential for flooding is also present. In this scenario, the flood causes instability in the electrical grid thus causing areas to be without electricity for an extended period of time. Below the scenario is shown in the way it will be presented step-by-step to the participants for the table top exercise.

Scenario 1: Flash Flood

1. The Neckar has reached the high-water level and has begun to destroy surrounding buildings, roads, the SWH substations and the transport routes.

2. The energy network is unstable due to the floods, and some areas of the city can no longer be supplied.

3. The first customers complain to the public on pages of social media e.g. Twitter, Facebook, etc.

The following scenario is considering a terrorist attack in Heidelberg.

Scenario 2: Terrorist Attack

1. Intelligence services receive information about a terrorist attack in Heidelberg on Christmas Eve. 2. Further information shows that the electrical grid is their target. 3. Three bombs exploded at the Pfaffengrund power plant, the substation south and north in

Heidelberg. The systems are destroyed and cause a breakdown of the power supply and the substation. Fortunately, there were no casualties.

Scenario 3: Cyber Security Attack

1. Stadtwerke Heidelberg IT & Energy notice higher energy consumption in the Bahnstadt district of Heidelberg.

2. It was confirmed by the Stadtwerke Heidelberg IT that the Smart Meter system was hacked, which led to an extremely high energy demand.

3. Hacking caused a power failure in all houses using smart meter technologies.

As part of the table top exercise the following team was created:

• two exercise coordinators o Professor from Heidelberg University of Applied Sciences o Leader of New Technologies at Stadtwerke Heidelberg

• one documentation coordinator


page 44 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

o research assistant for both Stadtwerke Heidelberg & Heidelberg University of Applied Sciences

• three observers o a research assistant at University of Wuppertal o innovation manager for Stadtwerke Heidelberg o regulatory manager

The exercise coordinators will lead the table top test. They will ensure that the discussions are relevant to the performance objectives of the exercise. The documentation coordinator will provide all paperwork before and after the exercise. The observers will take notes of the relevant discussions while keeping confidentiality of participants.

The following participants will be in roles at the Stadtwerke Heidelberg will be in attendance:

• Group Leader Planning Power Systems • Group Leader Central Management • Area Coordinator IT Infrastructure & Security • Staff Leader Occupational Safety • Department Manager Network Information • Staff Leader Company Communications.

These participants were identified, notified, and agreed to participate in the table top exercise.


page 45 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.3 CHARLIE (“Health care system”), Austria


Case study addresses the resilience of critical infrastructures in the context of health care systems as a whole. As a concrete example, the focus here on the Austrian health care system. Effective and cost-efficient treatments of patients require an increasingly complex coordination of several health care providers (HCP). These HCP are typically embedded in multiple formal and informal relationships with each other that result from their sharing of patients, information or practices. In the SmartResilience project it is there for imperative to consider that the resilience of a health care system can only be understood by focusing not only on individual HCP such as hospitals, but on the entire spectrum of HCP [6]. This spectrum ranges from primary care providers (first point of consultation), secondary care providers (medical specialists) and tertiary care providers (e.g. referral hospitals). Cost, quality, effectiveness, and resilience of care is consequently a property of the entire system of flows of patient through individual HCP on each of these levels [1]. Although the main focus of the case studies in the SmartResilience project lies on individual cities, we therefore decided to study the resilience of health care on a country-wide level Error! Reference source not found.

Relevant threats are cyber-attack, natural threats (e.g. urban flooding5) and breach of privacy, yet the methodology that is currently developed can be generically applied in all situations where a specific type of health care providers faces a sudden and unexpected increase in the number of patients to be treated within a given time span.


The case study CHARLIE studies the resilience of health care systems by focusing on an urban flooding event in Austria, which might result in a mass incident of wounded people. Not only may the flooding event lead to a large number of wounded people, but at the same time it may lead to a partial breakdown of the transportation infrastructure and therefore put additional stress on certain health care providers, since others might become practically inaccessible. In this scenario, an integrated perspective is taken on the healthcare system. The focus is particularly on how disruptions like urban flooding events disrupt the coordination between different healthcare providers with respect to flows of patients through the system, as well as the resulting cascading effects.


1. Infrastructure maintenance 2. Likelihood of hospitals becoming in-operational 3. Emergency management 4. Real-time monitoring 5. Regional preparedness 6. Communication 7. Emergency medical respondent team 8. Hospital accessibility 9. Real-time monitoring of patient flow during emergency response 10. Transport and evacuation


1. Infrastructure maintenance 1. How well is the critical infrastructure maintained?

2. Likelihood of hospitals becoming in-operational 1. What are the most likely medical needs of the population? 2. Population density in potential affected areas 3. What is the likelihood of the hospital needs to be evacuated?

3. Emergency management 1. Number of certified assessors

5 Threat considered in the D5.1 report [18].


page 46 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

2. Percentage of completed courses of dedicated personnel 3. What is the available capacity of the hospital? 4. Number of patients connected to the guidance system 5. Is there information about the public health procedures? 6. Alternates and backups for Electric power in place?

4. Real-time monitoring 1. Is there a syndromic surveillance system in place?

5. Regional preparedness 1. Are there any external reviewers of the emergency plan? 2. Does a regional plan exist? 3. What is the quality of the regional plan? 4. Does the quality of the regional plan correlates with training? 5. Are regional plans annually reviewed?

6. Communication 1. Is it defined who is in charge of the media communication?

7. Emergency medical respondent team 1. What is the local staff response time? 2. What is the insular area staff response time? 3. What is the state staff response time? 4. How much time does it take to locate the medical response team?

8. Hospital accessibility 1. Number of ambulances 2. How much time does it take to get ambulances to the affected area

9. Real-time monitoring of patient flow during emergency response 1. Number of staff available to treat patients 2. Number of patients that were assessed in triage 3. Number of patients assessed in an affected area 4. Percentage of electronically reporting (including handheld devices)

10. Transport and evacuation 1. Is there a plan for patients triage according to their urgency level? 2. What is the quality of the transport and evacuation plans? 3. Are there any external reviewers of the transport and evacuation plans?


• Number of patients that cannot be taken care of by an appropriate healthcare provider within a specified amount of time

• Percentage by which capacity of a given healthcare provider is exceeded • Combined available capacity of all healthcare provider of an appropriate type within the considered

region

5.3.3 Detailed description of stress-test scenario/s Many of the indicators and thresholds described above are computationally extensive and can only be estimated from data using advanced statistical techniques. Therefore, MUW is developing a computational tool to assist the stress test. In the following we give a description of the particular type of stress test scenarios that are implemented within that tool. Note that this is still work in progress and as such all of the following pictures show preliminary results that have the main purpose to illustrate the scenario.


page 47 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Figure 17: Map of Austria with the position of all healthcare provider. Each of the circles represents a general

practitioner (turquoise), medical specialist (orange) or hospital (dark blue).

Figure 17 shows a map of Austria with all healthcare provider as circles. The colors of the circles represent the type of the provider. A turquoise circle indicates a general practitioner, orange ones correspond to medical specialists and dark blue indicates hospitals. Based on medical claims data it is possible to identify the activity for each of these healthcare providers on any given day in the data.

Figure 18: Map of healthcare provider in Vienna for a given day on the data. The size of the circles indicates

the number of patients treated by the given provider.

This is shown in Figure 18 for Vienna, where the size of the circles correlates with the number of patients of the given provider within a given time span. Note that although the actual address of the providers is known, in these figures we only show them localized at a random position within their district for privacy reasons.


page 48 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Now a particular stress test scenario can be specified as follows. The scenario consists of a selection of health care providers that either (i) receive a sudden surge of an in-flow of patients of a specific type or that (ii) become in-operational (for instance all hospitals within a given distance to the river Danube). As a consequence, not only the sudden in-flow of patients has to be brought to suitable provider, but also the patients of those providers that have become inoperational. The functionality of the healthcare system is then a correlate of how fast and efficient the system is able to deal with this cascade of patient displacements within the healthcare system. In order to assess this ability of the healthcare system, we consider naturally occurring patient-sharing relations between providers. This is shown in Figure 19 for the federal state of Vorarlberg. On the left hand side, we show again the map of care provider and have selected one provider (highlighted by red arrow). The blue lines show how likely a patient that has encountered the selected provider has also encountered another provider of the same type. The thicker the blue line, the more likely that a patient has been treated by both of the provider that are connected by the line. This leads to a so-called patient-sharing network in which provider are connected by the number of patients that they share, see the right hand side for a visualization of the network with all links as shown to the left.

Figure 19: To assess the ability of the healthcare system to cope with a sudden and massive in-flow of

patients we evaluate how efficient the system is able to distribute flows of patients based on naturally occurring

So if the selected care provider exceeds its capacity or becomes inoperational, the most natural assumption is that the patients need to be taken care of by the provider that are close in the patient-sharing network. As they might exceed their capacity as a result of this in-flow, too, cascading patient displacements can be triggered that eventually span and overload the entire healthcare system of a given region – a critical point. Within this stress test, we evaluate how important each individual healthcare provider is bringing the healthcare system closer to this critical point.


page 49 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Figure 20: Based on this stress test, we assess the contributions of each individual healthcare provider to the

decrease in its functionality level. These contributions are visualized here for each provider as different colors, ranging from green (low impact on functionality level) over blue (medium impact)

to red (provider is critical for the functionality level of the health care system).

The result is that for each healthcare provider we can estimate from the data how much the functionality level of the healthcare system decreases if this particular provider becomes inoperational or exceeds its capacity. Preliminary results (with random positions within the district for each provider) are shown in Figure 20, where again each provider is a circle with a color that indicates the size of the cascade of patient displacements. The redder the color, the larger the decrease in functionality level (measured in numbers of patients which cannot be adequately taken care of within the district).

For the stress test, scenarios can be defined by the user through the selection of certain combinations of providers that receive a large number of patients or that become inoperational. The tool then computes the decrease in functionality level in the considered scenario and returns the corresponding indicators in a format such that they can be conveniently used for further analyses in the SCI dashboard.


page 50 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.4 DELTA (“Airport”): Air transport infrastructure, Hungary

5.4.1 Basic description of scenario

Embedded transportation facilities have a key role in today’s (still) mobility-based economy. Air transportation is the prominent transportation modality now. The significance of this transport modality and the main end-user needs related are discussed in D1.3 [14]. This is a critical infrastructure with increasing smartness on the passenger service side, at passenger terminals, for example electronic boarding passes, automated boarding and border control gates, intelligent CCTV surveillance are already present and biometric identity based boarding as well as passenger tracking are already introduced at key airports (e.g. Los Angeles Airport just announced biometric boarding passes). Such phenomena are changing the terminal as environment for the passengers, as a side effect, terminal staff is decreased, passengers should be more standalone in terms of navigating in the terminal. Smart solutions, like user-friendly local applications for mobile devices are offered by the airport or by airlines.

The Budapest Liszt Ferenc International Airport (BUD) is the largest international airport of the Hungarian capital city, Budapest. The total land area of the facility is 15,050,000 square meters. The facility has both commercial (passenger, cargo) and general aviation traffic, but is also occasionally serving military airplanes (e.g. KC-130s in Balkan wars). In 2016, the commercial aviation handled more than 11.4 million passengers, more than 90,000 aircraft movements and 112,000 tons of cargo with coordinated work of approximately 10,000 people. Border traffic size: 3,677,714 people did cross the Schengen external borders at the airport in 2015. There is a 5-level security check system, covering public and national security, passenger security, border security and customs checks. Blocking of the air traffic in the main airport of a member state has a deep effect over the country and related airports.

Smartness level of airports are defined by generations (Airport, Airport 2.0, Airport 3.0, Airport 4.0) as described in Table 7.

Table 7: Smartness level of airports

Airfield Basic infrastructure (runway, tower or remote tower). Limited facilities. Mainly self-operated private aircrafts.

Airport All about manual and analogic processes. Long lag-time between resource solicitation and the airport answer.

Airport 2.0 Implementation of self-service thanks to the automation of some key flow processing tasks (bag-drop, passport check).

Airport 3.0 Several focused initiatives to leverage digitalization so that to optimize flow monitoring and processing.

Airport 4.0 Full-connected with all stake-holders. Superior proactivity and reactivity to adapt to the real-time solicitation of the airport (operational needs, customer requests, etc.).

Most airport of the world are in stage 2.0, some of them already reached 3.0. Budapest Airport is now under development from Airport 2.0 to Airport 3.0, which is according to the state-of-the art of airport around the world. Airport 4.0 generation is forecasted by business strategists and expected to arrive by 2022.

BUD T1 terminal consists of two main areas, landside and airside, as illustrated in Figure 21. Landside is open for the public and can be visited by anybody. Airside can only be reached by authorized airport personnel or visitors in their escort, passengers with valid flight ticket and after passenger security check. The airside is physically separated from the landside with walls, security doors, checkpoints and – outside the building – a Smart Fence. The area is protected by the Airport Police Directorate and by the Armed Security Guard of BUD-Sec. CCTV surveillance is operational on both landside and airside.


page 51 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Terminal 1 and its environment:

Terminal 1 inside areas - 1st Floor:

Border between Airside and Landside (SRA checkpoint, line of passenger security check)

(Image: BUD maps, property of the airport operator)

Figure 21: Budapest airport terminal 1 division to airside and landside (other markings are representing shops, are from the original opera and not relevant for this deliverable)

Relevant types of threats include terror attack, cyber-attack and explosion (see e.g. D1.3 [14] and D5.1 [18]).


page 52 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.4.2 Functionality elements, Functionality indicators and limits (thresholds) The stress-test scenarios imposed on the SCI will lead to degree of loss of critical functionality measured by selected functionality indicators. Thresholds are established that reflects the acceptable levels of loss of critical functionality, given a stress. If the thresholds are exceeded, then the SCI has failed the stress-test, i.e. it is not sufficiently resilient against severe stress.


1. Passenger flow 2. Cargo operation 3. Departing flight time 4. Accident rate 5. Loss of income


1. Passenger flow 1.1. Numbers of passengers per time unit

2. Cargo operations 2.1. Cargo payload from bill of materials database (tons/flight) 2.2. Number of cargo flights take off (flights/hour)

3. Departing flight time 3.1. Number of flights/hour

4. Accident rates 4.1. Number of incidents registered per hour 4.2. Updated number of incidents (incidents/million passengers)

5. Loss of income 5.1. Assessed profit margins per hour 5.2. Total income from all activities (Euro/hour)


• The overall level of functionality should not be below 10 % of the nominal functionality level • The evacuation time should not be more than 20 minutes • The maximum allowed "loss of functionality" should not be more than 77 %



BUD T1 is a medium-size passenger terminal, at the day of the exercise it will have 1000 passengers, more or less randomly divided between airside and landside. Two threat scenarios will be played through, forcing the security personnel to perform evacuation accordingly.

The test environment will simulate a common day of operation of the terminal itself and handling the passengers, local staff and people other than passengers and their belongings. The situation will change suddenly and local evacuation plan will be executed short after the decisions made by the authority involved.

HNP has prepared two different stress-test scenarios (Case 1 and Case 2), with two sub-scenarios for Case 2, i.e. Case2A and Case 2B. The scenarios are described and illustrated below.

Figure 22 shows the possible deployment of the volunteers during the preparations of the case studies.


page 53 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

General site map:

Position and number of people to be evacuated


Figure 22: Possible deployment of volunteers

Stress-test scenario case 1:

Detected IED trace threat in the middle part of the terminal (passenger security), between airside and landside, in cascade with two other unattended luggage alerts (1st at the center, 2nd at arrival airside baggage claim area, 3rd at the mezzanine level). At the same time Disaster Management Center alerts HNP that their Cerberus server responsible for operating the terminal’s automated extinguisher system has been cyber-attacked exploiting vulnerability CVE-2012-2999, resulting in denial of service, meaning the automated fire extinguishing system is blinded and will not be operational in case of explosion or fire.

Therefore, sealing of terminal parts with only partial evacuation of the terminal – as usual in case of unattended luggage - is not enough, operational level of the facility has dropped, as without operating Cerberus PRO fire safety system it cannot operate safely. Full evacuation has to be commenced in two directions as the location of the IED threat cuts the terminal area in two, between airside and landside.

Passengers and staff will be evacuated in two main directions, as the central of the threat separates the territory in two parts. This is illustrated in Figure 23.


page 54 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Figure 23: Evacuation of passengers and staff away from threat – out of terminal

Direction of evacuation

Bomb



Stress-case scenario case 2A:

Industrial accident near the terminal. Sulphur dioxide is released into the air. Complete evacuation is not possible. The placement of persons must be solved at the terminal. The gas is self-dispersed. The expected duration of the pollution is approx. 5-6 hours. Some terminal parts are also contaminated, passengers have to be secured at safe terminal parts. Landside and airside apron areas also have to be empty. In this case there is no „pure” evacuation based on main threats, but emergency handling of passengers and staff will provide a challenge to the local authorities and to the airport operator. People in the forefront of the Terminal building and passengers at the airside have to be evacuated from the external parts into the building. This is illustrated in Figure 24.


page 55 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Figure 24: Evacuation of passengers and staff away from threat – into the terminal


Toxic cloud

Direction of toxic cloud




page 56 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Stress-case scenario case 2B:

Ongoing decontamination, several escape routes free and safe, terminal has to be fully evacuated. Passengers and staff will be evacuated in one main direction (towards the landside, e.g. public parking areas), as the cause of the blocked transport interferes all people within the area. This is illustrated in Figure 25.

Figure 25: Evacuation of passengers and staff – towards the landside


Contaminated area



Additional details about the DELTA exercise can be found in Annex 2.


page 57 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.5 ECHO (“Refinery”) - NIS refinery-industrial plant, Serbia


City of Pančevo has the so called Southern Industrial Zone located at the southeast edge of town, right next to the residential area of the city, app. 4 km away from the city center. In addition to the compound of the HIP-Petrohemija a.d. Pančevo, this zone includes the HIP Azotara Pančevo a.d. and NIS Oil Refinery Pančevo. The area is connected to road, rail and river circulation by means of the port on the Danube River. In this industrial zone, there is a production of petroleum products, basic chemical products, polyethylenes, mineral fertilizers, calcium ammonium nitrate, carbamide and NPK fertilizers.

Relevant threats are terrorist attack, natural threats and new technology related threats. Within ECHO case there is one scenario – BLEVE (Boiling liquid expanding vapour explosion) – identified as the major accident with the most severe consequences based on calculation and modeling done and presented in the Safety report. BLEVE effect refers to spherical vessel with capacity of 1000m3. There are many causes that could provoke such an event. One of the main causes that were analyzed is leakage of liquefied petroleum gas on the flange located next to the LPG vessel, with subsequent ignition of the LPG. Combustion of LPG that are in direct contact with the vessel will cause heating up of the LPG inside the vessel and this will lead to rapid vaporization and increasing the pressure, which will cause explosion. The consequences of such an explosion could lead to “domino” effects due to thermal radiation and debris. Such effects could be provoked by terrorist attack or even sabotage as well [18].


The stress-test scenarios imposed on the SCI will lead to degree of loss of critical functionality measured by selected functionality indicators. Thresholds are established that reflects the acceptable levels of loss of critical functionality, given a stress. If the thresholds are exceeded, then the SCI has failed the stress-test, i.e. it is not sufficiently resilient against severe stress.


1. Presence of fire fighters at the production site; • Number of fire fighters per shift at site;

2. Emergency response • % Emergency exercises completed according to the schedule;

3. Mitigation measures • Critical products /service available during an event?

4. Corrective actions • Time taken to complete corrective actions from incident investigation


• The overall level of functionality should not be below 10 % of the nominal functionality level

• The evacuation time should not be more than 20 minutes

• The maximum allowed "loss of functionality" should not be more than 77 %



page 58 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.5.3 Detailed description of stress-test scenario/s The Smart Industrial zone in the city consists of several infrastructures such as Azotara, NIS oil refinery, HIP, etc. These infrastructures are interconnected and interdependent. The whole zone is regarded as one big CI or CIs.

Within ECHO case there is one scenario – BLEVE (Boiling liquid expanding vapour explosion) identified as the major accident with the most severe consequences based on calculation and modeling done and presented in the Safety report. From the Safety report, two worst cases have been considered for BLEVE effects.

• Case 12 from the Safety report– Spillage of propylene from sphere reservoir FB-16705 (1000m3) on the warehouse 16700 Description of the incident: Propylene spillage occurred on the first flange joint of the pipeline for decanting of propylene from the sphere FB-16705 due to propulsion of the seal of 50 mm in length and 3 mm in width. Time duration of the incident – 30 minutes.

• Case 14 from the Safety report: Incident on the FB-5001 vessel (10m3) for hydrogen reserve Description of the incident: Since reservoir with hydrogen is under high pressure, it may come under its total destruction physical explosion under certain conditions. When a vessel containing pressurized gas breaks, as a result of that, stored energy is released. This energy may produce shock wave and flying fragments of vessels. Time duration of the incident: 1 minute.

Besides this reservoir FB-16705 (1000m3) there are 5 more reservoirs with the same capacity, on which potentially BLEVE effect could happen. There are many causes that could provoke such an event.

One of the main causes that was analyzed is leakage of liquefied petroleum gas on the flange located next to the LPG vessel, with subsequent ignition of the LPG. Combustion of LPG on the affected valve will have direct contact with the vessel will cause heating up of LPG inside the vessel and this will result in rapid vaporization and increase pressure which will lead to explosion. The consequences of such explosion could include only very hypothetically a “domino” effect due to thermal radiation and flying debris. Such an effect could be provoked by terrorist attack or sabotage (See the reference: SmartResilience (2017), D 5.1 report [18]).

Hypothetically, it is also possible to get affected by cyber-attacks on ICT systems.


page 59 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.6 FOXTROT (“Water”) - Drinking water supply in Sweden


Drinking water is often called the most important food. Sweden has for long enjoyed top quality drinking water straight from the tap. With an average daily household consumption of 160 L/person, drinking water is used for plenty of other purposes than just drinking, and the water quality is taken for granted by the majority of the Swedish population. There is an overall good state of the art relative to other critical infrastructure and other European countries:

• Overall plenty of supply

• Overall good quality

• Good knowledge of risks

• Robust system for monitoring vulnerabilities and risks

In recent years, however, a number of threats (now and in the future) towards the Swedish drinking water have appeared on the horizon. These threats include an increasing water shortage in some areas, climate change, which can increase the risks of pathogens in the water, and unexpected events such as heavy rainfall that contaminate the sources for drinking water.

In the FOXTROT case, the focus is on the microbial contamination of water and threats to the ICT systems in the waterworks and distribution. Waterborne disease is linked to climate change and is predicted to be more frequent in the future as described in each threat description. Threats to the ICT-systems is linked to the vast digitalization of industrial processes, including drinking water production, causing new challenges for the producers to prevent vulnerabilities in the systems [18].


The stress-test scenarios imposed on the SCI will lead to degrees of loss of critical functionality measured by selected functionality indicators. Thresholds are established that reflects the acceptable levels of loss of critical functionality, given a stress. If the thresholds are exceeded, then the SCI has failed the stress-test, i.e. it is not sufficiently resilient against severe stress.


1. Produced drinking water 2. Water delivered to end users 3. Storage level in water towers


1. Produced drinking water 1.1. Volume per second of produced water 1.2. Time in hours of total loss of production

2. Water delivered to end users 2.1. Percentages of end users connected to the water distribution that receives drinking water. 2.2. Quality of the drinking water meets the standards

3. Storage level in water towers 3.1. Height of water level in storage towers


1. The overall level of functionality should not be below 80 % of the nominal functionality level 2. The total production loss should not be more than 72 hours 3. The water delivered should be of good quality 4. The maximum allowed "loss of functionality" should not be more than 50 %


page 60 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

A final set of functionality elements and functionality indicators will be selected and corresponding threshold values will be established prior to the exercise.


The scenario takes place in a medium-sized Swedish city with 10-15,000 inhabitants. In the city there is a waterworks that supplies approximately 10,000 people with drinking water. The water is taken from a surface/ground water source and is after the purification process distributed out to the people in the city.

It has been raining for about a week. At the beginning of the week about four to five millimeters a day and the soil begins to get saturated, the rain does not decrease but increases and after seven days it has reached about 40-50 mm. Then the city is suffering from an intense rainfall, a so called 100-year rain, and for about 24 hours it rains intensively. A total of about 150 mm falls on an already saturated field. SMHI (Swedish and Meteorological and Hydrological Institute) goes out with warning class 36. The heavy rain is leading to floods. All low points in the area, such as road tunnels, are flooded and so are many basements.

This leads to the following:

• One of the wastewater treatment plants have suffered loss of electricity and has lost its functionality and automatic controls. The surface water source is flooded. There is no electricity the treatment plant and it faces a risk of by-pass because of the large amounts of water still coming in.

• Fecal residues are coming into the raw-water for drinking water production with the flood. But also via leakage of wastewater into the drinking water distribution network.

• A number of pipelines in the distribution network have been demolished, of which two large pipes (200 cc) are completely destroyed. This leads to water towers empty quickly.

• After a time also the waterworks SCADA system goes down due too loss of electricity.

6 Very extreme weather is expected which could pose a great danger for the public and very large disturbances in

important social functions. The public is advised to follow new information on the internet, radio or television.


page 61 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.7 GOLF (“Transport”) – Flooding in the City of Cork, Ireland


Cork City, located at the head of a tidal estuary and the downstream end of a large river catchment is prone to both tidal and fluvial flooding. Cork City is the second largest city in the Republic of Ireland with a population of 125,622 as per the 2016 census. Flooding is the main threat in the GOLF case study. We selected the critical infrastructures for this case study based on the likelihood that tidal and fluvial flooding events would disrupt them. In the project proposal, we have identified the following critical infrastructures as the most relevant for Cork City: public transport and water supply [18].


The stress-test scenarios imposed on the SCI will lead to degrees of loss of critical functionality measured by selected functionality indicators. Thresholds are established that reflect the acceptable levels of loss of critical functionality, given a stress. If the thresholds are exceeded, then the SCI has failed the stress-test, i.e. it is not sufficiently resilient against severe stress.


1. Road access impacted by urban drainage capacity 2. Water reservoir storage


1. Road access impacted by urban drainage capacity 1.1. Precipitation 1.2. Water discharge at the Inniscarra Dam

2. Storage in water reservoirs 2.1. Levels at which customers loose water supply

Examples of thresholds may be:

1. Maintain 24 hours water supply from water reservoirs, loss of water supply at depth level x 2. Precipitation levels in mm/m2 at which absorption capacity of urban drainage scheme is reached

and roads start to flood disrupting public transport routes 3. Redundancy in public transport network to reroute and maintain service level

Three main thresholds can be identified:

1. Incoming water amount below drainage absorption capacity (protection) 2. Incoming water amount above drainage absorption capacity but below flood gate, pump capacity at

maximum and available sandbag/capacitive absorption level (defense) 3. Incoming water amount above defense capacity (breaking point for SCI GOLF)

Threshold is measured with use of the following elements and indicators:

1. Amount of incoming water: 1.1. precipitation in mm/m2 1.2. run-off (Q) in m3/h) 1.3. floodometer in cm 1.4. floodometer in riverbank load percentage%

2. Absorption capacity 2.1. drainage maximum capacity (m3/h)

3. Defense ability 3.1. flood gate capacity (m3/h) 3.2. pump capacity (m3/h)



page 62 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

t₀ The urban drainage system has 100% capacity

t₁ Still the urban drainage system is at 100% capacity

t₂ Excessive precipitation bring the urban drainage system gradually to 0% capacity

t₃ The urban drainage is non-functional, flooding builds up in the streets

t₄ The precipitation rate goes down, the drainage system starts to recover

t₅ The plant running at 100% of the capacity


There are two stress-test case scenarios: Fluvial flooding and pluvial flooding.

Scenario 1: Fluvial Flooding Event

Met Éireann has issued a Code Red weather warnings, and the daily rainfall of 120mm has fallen on the Lee catchment area for the past three days. Consequently, this has led to a dangerous build-up of water levels at the Inniscarra dam. To protect the dam structure, the electricity utility has increased the volume of discharge to over 500 cubic metres per second. This increased discharge volume is leading to severe flooding on the western side of Cork City and is the level at which we expect Major Flooding.

We expect there will be extensive flooding of properties and businesses and a major threat to life. There will be widespread flooding of road/rail networks. There will be a major disruption to traffic. Evacuation is likely to be required. The following agencies are notified: (a) Cork City Council; (b) An Garda Síochána (Police); (c) Civil Defence; (d) Fire Service; (e) Health Service Executive; (f) Voluntary Emergency Services & local organisations; (g) Defence Forces (Army); (h) Irish Coastguard.

The Flood Response Group has decided the situation require the activation of the Major Emergency Plan.

The water treatment plant on the Lee Road has malfunctioned due to flooding, and water supplies at the reservoir are diminishing. Access to the Mercy Hospital in the City Centre is not possible, and the Accident and Emergency department has been closed. 30% of the city is without water supply, and most of this area is also without electricity [18].

Scenario 2: Pluvial Flooding Event

A combination of high tides due to surges and high river levels are leading to pluvial flooding as surface-water cannot drain to the river because of high water levels in the receptor. As a result, drains are becoming surcharged leading to the risk of localized flooding of streets and property, and there is also the risk of manhole covers being lifted and displaced by pressure build-up in the drains, which in turn leads to a health and safety risk.

The council have closed off the affected road to transport and pedestrians. Crews are available to clean up and hose down the soiled areas once river level drops. Water pressure in the water network is being maintained to ensure no contamination of the water network [18].


page 63 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.8 HOTEL (“Energy”) - Energy supply infrastructure in Helsinki, Finland

5.8.1 Description of SCI and most relevant threats The energy supply infrastructure in the city of Helsinki, mainly through the city-owned energy company HELEN Oy, is producing and distributing three main products, district heating, district cooling and electricity.

Of these, electricity supply involves fair redundancies by alternative feed sources, and as the yearly mean temperature in Helsinki is +6°C and the mean for the warmest month +18°C (July, with average daily maximum of 21,5°C), cooling is less critical than heating. The main supply sources of district heating are the three major power stations, supported for peak loads by smaller heating plants and few limited connections to the adjacent systems of the neighboring cities. The distribution piping system for district heating in Helsinki is more than 1300 km in length and runs underground to all public and most private buildings in the city. District heating can easily cope with usual and many unusual winter circumstances, as the mean daily minimum temperature (-7°C) of the coldest month (February) is well above the designed minimum temperature for the system [12]. However, the system is not designed to cover sustained load approaching the record cold in Helsinki (-34°C in 1987). In spite of the redundancies, disturbances in critical or multiple nodes of sourcing and distribution could also limit the heating function during severe cold spells. Such challenges represent threats to the heating system function, customer expectations, and in extreme cases, to public health. The relevant threats for the case are natural and new technology related threats.

The case focuses on a scenario that can significantly disrupt the district heating supply or distribution in Helsinki, namely fire in the underground fuel storage of a power plant (supply node for district heating).

Short-term events can trigger the scenario that then will take some time to develop significance, depending on weather and functionality of other resources in the system. There is experience on the initiation and management of the scenario, as self-heating and auto-ignition in solid fuel storage is not very rare [13]. More challenges could appear with e.g. increasing share of biomass-derived solid fuels in storage.

5.8.2 Critical functionality, functionality indicators and thresholds The stress-test scenario will result in a (recoverable) reduction of the critical functionality that is measured by the functionality indicators. The functionality limits are on one side reflecting the expected technical qualities of the SCI and other Cis involved, and on the other hand the details of the chosen scenario as it evolves and escalates in time.

The suggested functionality elements and corresponding functionality indicators for the case are:

1. District heat production and supply (element) 1.1. Production as percentage of full capacity (indicator, district heating output of Salmisaari

plant) 2. HSE performance (element, high priority on safety)

2.1. Percentage of working hours of relevant personnel (indicator on safety) 3. Interconnectedness in heat supply (element)

3.1. Production as percentage of full capacity of another plant (indicator, output of another plant) 4. Number of high priority people supplied with district heating (element)

4.1. Number of affected high priority people in case of system cooling (indicator) 5. Economic performance (element)

5.1. Revenues as percentage of those when operating at full capacity (indicator).

In the scenario, the functionality and time limits (thresholds) are described in the event flow as shown below.

5.8.3 Detailed description of stress-test scenario The scenario of self-heating and auto-ignition typically involves a bed of solid fuel, where the combination of air ingress, limited heat transport and exothermal oxidation may allow for gradual heating up to the ignition of a smouldering fire, especially in cases of long storage time, sensitive fuel quality and/or warm fuel entering storage. Storing fuel in underground silos will add to the challenge by limiting access for countermeasures.


page 64 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

Significant disruption of district heating service under high load (cold spell) would normally also require additional trouble from failing alternative fuel feed and loss of another heating plant source.

In a severe but conceivable case, auto-ignition and fire could initiate and develop as follows: a batch of fuel has entered the storage silo when warm, or with a heating and ignition-prone composition/quality, or was stored for a lengthy time in the silo. During a midwinter cold spell (below -20 to - 25°C) at approximately full load of the district heating network, after ignition the thermal gradient and gaseous smouldering combustion products diffuse only slowly from of the fuel bed centre. When the period between ignition and detection is not short, a relatively extensive volume of fuel will be involved in the resulting smoldering fire. The adopted extinguishing countermeasures include water and possibly nitrogen injection, and directing the silo contents to boiler for removal. Nevertheless, the fire in this case will proceed to emit combustion gases also to the area surrounding the plant, annoying the residents. Public complaints in local and national media may result in untimely technical modifications. Meanwhile, heated fuel directed to combustion will result in conveyor damage so that all silos become unavailable. With no support oil fuel due to its high viscosity in a cold storage tank, the backup fuel comes by truck transport from another plant, but at a low rate meaning plant derating to 60% capacity. Then boiler failure takes a nearby heating plant out of service, and the combined deficit in production results in gradual system cooling in the western part of the network. The impact becomes significant during the extended cold spell and extended period of reduced plant capacity. Following timeline is proposed for the scenario.

t₀ The Salmisaari plant is running at 100% of the capacity

t₁ Still the plant is running at 100% of the capacity

t₂ Fire in one silo takes out conveyor belt and thereby all underground fuel silos. Attempts to get replacement fuel from the plant storage fail. Production of the plant falls to 60% of the capacity, relying on fuel transported from Hanasaari plant by truck. A nearby heating plant falls to 0% production, so that the western district of the heating network starts to cool.

t₃ The storage is non-functional, Salmisaari plant continues to operate at 60% capacity and the nearby heating plant remains out of service. The western district of the network has cooled so that it triggers evacuation of high priority people from elderly peoples’ homes.

t₄ The conveyor belt has been repaired and the Salmisaari plant becomes functional at 100% of the capacity.

t₅ The nearby heating plant is also recovered and running at 100% of the capacity, so that all parts of the western district are supplied with heating as required by the demand.


page 65 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

5.10 INDIA (Integrated case) The integrated case deals with the situation when critical Infrastructures are located in close proximity: in the same town or city or right next to each other. The most basic example for this is a transportation hub next to a factory, but there are more complicated networks around the globe. Increasing smartness level of critical infrastructure do not change this fact, however enables better interconnectedness on information and management level. This allows us to harvest data from different Smart Critical Infrastructures (SCIs) in a same geographical location, analyze it and provide not only a resilience or functionality assessment for the individual SCIs, but for the whole area as well. Such is necessary because the physical proximity of the SCIs to each other means that an adverse effect, like an industrial accident may not only cause economical interaction as described in D2.3, but damage the other SCI, possibly causing a new disaster, we call this ‘ripple effect’. Another way of interaction is when two or more SCIs are hit by different disasters, but at the same time or in a short time period, in the same geographical area. For example, heavy raining causes flood the next day after the industrial accident, hitting the underground storage, but also the airport which is not even recovered from the industrial disaster, we call this cascade effect. In the INDIA case we perform a tabletop exercise by going through a fictitious, integrated test case built from the previous seven case studies.

5.10.1 Description of fictitious city (multiple SCIs) The integrated case study INDIA covers SCIs from case study ALPHA to HOTEL. The cascade and ripple effects of different negative impacts on SCIs are presented in a single fictitious geographical area. BRAVO town is a Smart City with a population of approx. 12.000 people in 1.260 buildings in a total area of 116 hectares. There is a branch office for a key national financial system operator, the ALPHA bank in the city. Health services are provided by a small hospital and GP districts operated by the national health system CHARLIE. The town is built next to the GOLF river, on its right bank. Upstream is the FOXTROT reservoir, on the left bank is the ECHO industrial production site. Next to the industry lies airport DELTA. Downstream the right bank there is a fully automated underground coal storage operated by HOTEL Inc.

The fictitious city, involving multiple SCIs, is illustrated in Figure 26.

Figure 26: Cascading effects among multiple SCIs in fictitious city

The elements of the fictitious city are put together from the existing geographical locations of the case studies.


The main logic behind integrated case study INDIA is the theory on cascade and ripple effects caused by interdependency within SCI (level 2 in the theoretical assessment model of Smart Resilience project), in a given geographical area (the level 1 in the theoretical assessment model defined in the resilience assessment methodology [16]. There are several types of interdependencies, most of these were analyzed and explained

http://smartresilience.eu-vri.eu/sites/default/files/publications/SmartResD3.2.pdf


page 66 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

in the report on interdependencies and cascading effects of smart city infrastructures [15]. The rest of case studies can analyze all but the geographical type of interdependency, in which case a localized event might affect the state of infrastructure systems that are in proximity. A fictitious “BRAVO City” was established to bridge this gap and enable the project to analyze functionality changes emerging from ripple effect transferred from one SCI to the other because of they are in located in the same area of effect. A very simple example is, that it does not really matter how resilient the airport is if there is a nuclear plant melting down right at next door.

Therefore, case study INDIA does not have different functionality elements and indicators other than the case studies A-H, but with a comprehensive approach where parallel but not linked (cascade) and interdependency-caused (ripple) effects appear for different SCIs in a geographically linked context. This approach will support development and validation of methods for functionality level measurement of a given geographical area, in this case, the fictitious BRAVO city. In order to enable this, functionality threshold is set to 1 on a scale of 1 to 5, according to theoretical measurement defined in the resilience assessment methodology [16]. The figure below is only a depiction on how an INDIA functionality level assessment can be visualized. Each case study has to be assessed for functionality for each phase, taking into account the effect they can release on each other. For example, an industrial accident from a nearby refinery causes microbial infection in drinking water supply, overburdens the health system and forces the airport to close. The landscape visualization allows a comprehensive view: one can see the functionality level of an individual level and the functional level of the whole at the same time.

Figure 27: Comprehensive visualization of FL change

5.10.3 Detailed description of stress-test scenario/s Integrated test INDIA is a tabletop simulation, consists of a series events related to case studies ALPHA to HOTEL.

1. ECHO (explosion at production site) 2. CHARLIE (number of injured is over the caps of local health facilities, transportation concerns arise) 3. DELTA (due to dense smoke and microparticles, air transportation has to stop, and airport has to be

evacuated) 4. GOLF (heavy rain causes pluvial and tidal flood) 5. FOXTROT (drinking water supplies suffer microbial infestation as rain washes carcasses into water)




page 67 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

6. HOTEL (underground storage is at risk of flooding, storage sealed but self-ignition strikes, power plant shuts down)

7. BRAVO (community has to cope with lack of infrastructure) 8. ALPHA (a faulty inbound transformation releases an EMP through the electronic system)

Figure 28 shows the event map of the INDIA case, starting from above, with the accident in ECHO. Red lines represent immediate ripple effect of physical nature from ECHO case: victims flow to the hospital, airports are locked down by the smoke etc.) and is numbered 1. Second wave effects are numbered 2 and are representing the second wave of ripple effects, which is not direct physical effect but consequences of physical co-location and interdependency (e.g. small mammals died because of released toxic material gets washed into the drinking water reservoir), and are arising a short while after. The same applies for the rest of arrows and numbers, each number represents a wave of events, direction of the arrows represent the origin and the target SCI.

Figure 28 visualization of the event chain and affected SCIs

Based on the event map above, we set up a story which will be presented to the tabletop participants and are summarized as follows.

„Human error leads to explosion in the industrial plant, there are more injured than the local hospital could handle, dense choking smoke cripples air traffic, injures humans and kills smaller animals. In the evening, heavy rain starts, resulting in pluvial flood, a tidal flood is imminent as wave is coming downstream. Drinking water reservoir gets infected with microbial threat as carcasses are washed into the water by the rainfall. The coming flood endangers the underground storage which has to be sealed off, but then, a self-ignition hits the facility. As the town switches to inbound electricity through national power grid, a faulty transforming station releases an EMP shockwave on the electrical system of the ALPHA bank. Following the disaster, a series of insurance frauds are shadowing the recovery.”

The tabletop simulation will be facilitated by test coordinators from BZN and performed in parallel together with the Hungarian National Public Service Exercise of 2018, whose topic is critical infrastructures under cyberattack and called Integrált Védelem 2018 (Integrated Defense 2018). The exercise will be held on 23rd and 24th of April, 2018.

Test players will not gather for the INDIA simulation, they will contribute from their own workstations.

Reservoir Water supply infrastructure

Reserves

ECHO:Production

Emergency services

CHARLIE:Health Care

Traffic lights

Transport infrastructure

DELTA:Transportation (airport)

Number of passengers

Power plants

HOTEL:Energy supply

FOXTROT:Water supply

GOLF:Government (flood)

Emergency servicesCooperationReserves

Financial infrastructureOnline banking

ATMInsurance companies

ALFA:Financial system

Underground storage

Storage

Level of toxicity

Crisis management

Power suply

1

Medicine supply

ACCIDENT:man-caused

release of toxic aromatic liquids

21

1

2 2

3

4

4

5

Drinking water supply

BRAVO:Smart City

5


page 68 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

References

[1] Avdeeva, Y., van Gelder, P. (2014). Deliverable 6.1: Stress test methodologies. EU project INFRARISK, Project No. 603960 (2013-2016), Contact: ROD, Ireland.

[2] Barnett ML, Christakis NA, O’Malley AJ, Onnela J-P, Keating NL, Landon BE, Physician patient-sharing networks and the cost and intensity of care in US hospitals. Med Care, 2012; 50(2): 152-60.

[3] Esposito, S., Stojadinovic, B. (2016). Deliverable 5.4: Report on strategies for stress test implementation at community level and strategies to enhance societal resilience using stress tests. EU project STREST, Project No. 603389 (2013-2016), Contact: ETH, Zurich, Switzerland.

[4] International organization for Standardization (2012). ISO Standard 22300:2012 Societal security -- Terminology, Geneva.

[5] International organization for Standardization (2012). ISO Standard 22398:2013 Societal security – Guidelines for exercises, Geneva.

[6] Krauß, M., Berg, H.P. (2011). New Evaluation of External Hazards in the Light of the Fukushima Accident. 9th International Probabilistic Workshop, Budelmann H., Holst A. and Proske D. (Eds.), Braunschweig, Germany.

[7] Kröger, W. (2014). Resilience of Critical Energy Infrastructure. In Thoma, K., Resilien-Tech "Resilience-by-Design": Strategie für die technologischen Zukunfsthemen, Germany.

[8] Landon BE, Keating NL, Barnett ML, Onnela J-P, Paul S, O’Malley AJ, Keegan T, Christakis NA, Variation in patient-sharing networks of physicians across the United States. JAMA, 2012; 308(3): 265-73.

[9] Mieler, M., B. Stojadinovic, R. Budnitz, M. Comerio and S. Mahin, 2015. “A Framework for Linking Community-Resilience Goals to Specific Performance Targets for the Built Environment”, Earthquake Spectra, vol. 31, no. 3, pp. 1267-1283, August.

[10] Mignan, A. (2014). Deliverable 2.4: Report on lessons learned from on-going and completed EU projects. EU project STREST, Project No. 603389 (2013-2016), Contact: ETH, Zurich, Switzerland.

[11] Pham HH, O’Malley AS, Bach PB, Salontz-Martinez C, Schrag D, Primary care physicians’ links to other physicians through medicare patients: the scope of care coordination. Annals of Internal Medicine, 2009; 150: 236-42.

[12] Pirinen P et al. (2012) Climatological statistics of Finland 1981-2010. FMI. https://helda.helsinki.fi/bitstream/handle/10138/35880/Tilastoja_Suomen_ilmastosta_1981_2010.pdf?sequence=4

[13] Sipilä J. Emerging risk issues in underground storage of bituminous coal. Aalto University, Helsinki 2013, 148 p.

[14] SmartResilience (2016). Deliverable D1.3: End users’ challenges, needs and requirements for assessing resilience, EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany. http://www.smartresilience.eu-vri.eu/sites/default/files/publications/SmartResD1.3.pdf, accessed on October 8, 2017.

[15] SmartResilience (2017). Deliverable D2.3: Report on interdependencies and cascading effects of smart city infrastructures, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany. http://smartresilience.eu-vri.eu/sites/default/files/publications/SmartResD2.3.pdf

[16] SmartResilience (2017). Deliverable D3.2: Assessing resilience of SCIs based on indicators, EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany. http://www.smartresilience.eu-vri.eu/sites/default/files/publications/SmartResD3.2.pdf, accessed on September 25, 2017.

https://helda.helsinki.fi/bitstream/handle/10138/35880/Tilastoja_Suomen_ilmastosta_1981_2010.pdf?sequence=4

https://helda.helsinki.fi/bitstream/handle/10138/35880/Tilastoja_Suomen_ilmastosta_1981_2010.pdf?sequence=4

http://www.smartresilience.eu-vri.eu/sites/default/files/publications/SmartResD1.3.pdf




page 69 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

[17] SmartResilience (2017). Deliverable D3.6: Guideline for assessing, predicting and monitoring resilience of SCIs. EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany. [In progress]

[18] SmartResilience (2017). Deliverable D5.1: Report on the results of the interactive workshop. EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany.

[19] Stadtwerke Heidelberg (2016) Integrated Annual Report, Stadtwerke Heidelberg. Germany [20] Stadtwerke Heidelberg (2016), Profile, https://www.swhd.de/Unternehmen, accessed on April.

25, 2017. [21] Sun L., M. Didier, E. Delé and B. Stojadinovic. (2015). Probabilistic Demand and Supply Resilience

Model for Electric Power Supply System under Seismic Hazard. European Safety and Reliability Conference ESREL, Zurich, Switzerland. 7-10 September.

[22] US Department of Homeland Security (2015). The Future of Smart Cities: Cyber-Physical Infrastructure Risk. https://ics-cert.us-cert.gov/sites/default/files/documents/OCIA%20-%20The%20Future%20of%20Smart%20Cities%20-%20Cyber-Physical%20Infrastructure%20Risk.pdf

[23] van Gelder, P., van Erp, N. (2016). Deliverable 6.2: Stress test framework for systems. EU project INFRARISK, Project No. 603960 (2013-2016), Contact: ROD, Ireland.

https://www.swhd.de/Unternehmen

https://ics-cert.us-cert.gov/sites/default/files/documents/OCIA%20-%20The%20Future%20of%20Smart%20Cities%20-%20Cyber-Physical%20Infrastructure%20Risk.pdf




page 70 Smar

tRes

ilien

ce –

D5.

2: S

tres

s-te

st a

nd e

valu

atio

n fr

amew

ork

A N N E X E S


page 71

DCL procedure

The DCL procedure consists of 4 pages, starting on the next page.


page 72

Responsible Related document/Description

Due Before

Login and start SmartResilience Indicator DB

RPP

RPP

RPP

RPP

RPP

Steinbeis Advanced Risk TechnologiesEuropean Virtual Institute for Integrated Risk Management

Start

As needed by the project

partners

Steps of the procedure

Procedure: Step 1 – Creating a dynamic checklist (DCL) for resilience asessement in the SmartResilience database (DB)

CI: Critical Infrastructure; DB: Database; DCL: Dynamic checklist; RI: Reslience Indicator; RPP: Responsible Project Partner

Click on “Scenario Dynamic checklist”

RPP

Add “new Resilience Index level”

Fill/edit the form with basic information (DCL name, select

type of CIs and threats, description)

Click on save

Is the information

correct?

Yes

No

Select the relevant scenario

RPP

Task 5.1 description of

scenario

Task 4.1 report

(chapter 4.5) and Task 4.4

common protocol

RPP

Does the scenario name

exist?

Click on the dropdown menu to find your scenario name

Yes

No Click on “+” to add new scenario

Follow procedure for creating

new scenario in

DB

End of Step 1

A-2

Page 1 of 4

RPP

Continue from the

procedure for adding

new scenario


page 73


Due Before

RPP

RPP

RPP



partners


Procedure: Step 2 - Creating a dynamic checklist for resilience asessement in the SmartResilience database (DB)

RPP

RPP Search issues in the additional filtering options (enter search

item, choose phase)

Drag and drop the issues to relevant phases

Yes

NoAre sufficient issues

selected?

Click on next

Yes

Does the issue exist

in DB?

No Add new record

Follow procedure for

entering issues in DB

Task 4.1 report

(chapter 4.5) andTask 4.4 common protocol

RPP

A-3

End of Step 2

Page 2 of 4

RPP

View/search in the recommended or relevant list of issues

Does the issue exist in

Recommended or relevant

list?

Yes

No

A-1

DB: Database; DCL: Dynamic checklist; RI: Reslience Indicator; RPP: Responsible Project Partner

Task 4.1 report


page 74


Due Before

RPP

RPP

RPP



partners



RPPSearch RIs in the additional

filtering options (enter search item, choose phase)

Drag and drop the RI to relevant issue

Yes

NoAre sufficient RIs

selected?

Click on next

Yes

Does the RI exist in DB?

No Click on “+” to add new

recordFollow

procedure for entering

RI in DB

Task 4.1 report

(chapter 4.5) and Task 4.4 common protocol

RPP

RPP

A-4

End of Step 3

Page 3 of 4

RPP

View/search in the recommended or relevant list of RIs

Does the indicator exist in Recommended

or relevant list?

Yes

No

A-2

DB: Database; DCL: Dynamic checklist; RI: Reslience Indicator; RPP: Responsible Project Partner

Task 4.4 common protocol(stage 2)


page 75


Due Before

RPP

Program

RPP



partners



RPP Click on finish

Generate results

Page 4 of 4

Fill out the minimum, maximum, and actual values of RI(weights optional)

Are all the values filled?

Yes

A-3

Click on “Save

changes”No

End

RI: Resilience Indicator RPP: Responsible Project Partner

Task 3.6 guideline (chapter 2.3.3.1)


page 76

Details of the DELTA exercise (example)

This annex provides some details about the planning of the DELTA exercise, which can be used as information for the other stress-test cases. However, detailed plans from the other cases are not required as input to this stress-test framework report. Plans may change rapidly; thus, updated stress-test plans should be kept within each case study task 5.3-5.11, without a need for updating this framework report. Development and updating of the stress-test plans are the responsibility of the case study task leaders.

Master event list (script)

Stress-test scenario case 1

Phase No.

Time Event description Responsible

1 10.00 HNP HQ commands the resilience assessment of T1 to prepare for the summer traffic peak.

HNP HQ

2 10.02-10.30 Participants and observers get their places in the Terminal building Test coordinator

3 10.31 Passenger security detects explosive traces in a bag. Role players

4 10.35 Cleaning personnel and local police patrol report unattended luggage next the exit corridor of baggage claim area.

Role players

5 10.36 Passenger reports unattended luggage at landside. Role players

6 10.40 Duty officer executes emergency evacuation protocol after reporting the facts to the Commander.

Role players

7 10.40-11.20

Police forces proceed the evacuation on the site, until airside and landside terminal parts are reported to be “clear”. Airport staff also have to leave the areas. No one remain in the terminal building, police secure all exits.

Role players

8 11.21 EOD Team begin to work. Test Coordinator

9 11.30-12.00 T1 cleared, restoring operation

10 12.00-12.05 Closing the phase

11 12.05-12.50 Break


page 77

Stress-test scenario case 2A

Phase No.


1 13.00 HNP HQ commands the resilience assessment of T1 again. HNP HQ

2 13.02-13.30 Participants and observers get their places in the Terminal building Test coordinator

3 13.31

Local Police HQ is informed about industrial explosion which results in ash cloud blocking air traffic, passengers stranded on the terminal exceeding capacity.

Duty officer executes emergency evacuation protocol but full evacuation not possible due to contamination of surrounding area.

Role players

4 13.32 Police forces execute the order. Role players

5 13.36 Airport staff report the possible contamination of terminal parts to the police patrol during the evacuation of passengers from the tarmac.

Role players

6 13.37-14.10 Police forces proceed the evacuation on the site, until airside and landside terminal parts are reported to be “clear”. All passengers present in the safe zone designated in the terminal airside area.

Role players

7 14.11 Closing the phase (proceeding to Case 2B) Test Coordinator

8 14.22-14.45 Time gap

Stress-test scenario case 2B

Phase No.


1 14.45 HNP HQ commands the resilience assessment of T1 again. HNP HQ

2 14.50

Local Police HQ is informed about ongoing decontamination, several escape routes free and safe, terminal has to be fully evacuated.

Duty officer executes emergency evacuation protocol.

Role players

3 14.52 Police forces execute the order. Role players

4 14.53-15.30

Police forces proceed the evacuation on the site, until airside and landside terminal parts are reported to be “clear”. All passengers present in the terminal forefront landside areas. Only airport staff remain in the terminal building.

Role players

5 15.30-16.00 Terminal is fully decontaminated Test Coordinator

6 16.00-17.00 Restoring T1 to operation Role players

7 17.00-17.20 Closing the break Test Coordinator

8 17.20-17.30 Break Test Coordinator

9 17.30-18.30 Post-exercise de-briefing Test Coordinator


page 78

Test personnel tasks related to use of the system

Local Coordination Team

• Exercise Coordinator: Lt. Col. Gábor Csapó, JD (HNP) • Documentation Coordinator: Lt. Col. Margit Pelikán Szőkéné (HNP) • Controllers (organizers): János Jelenek (HNP) and 28 officers (HNP) • Evaluators: Gabriella Oszlács (HNP), Zoltán Nagy (BZN), Dénes Perényi (BZN), László Árvai (BZN) • Exercise Safety Officer: Zoltán Székely, JD, Project Security Officer (BZN)

Test Support Unit

• Players in security role: played by professional airport security personnel, performing evacuation according to emergency evacuation plans.

• Players in passenger role: played by 1000 student of the National University of Public Service, they will perform randomly distributed (drawing cards) roles, like calm guy, go with the flow, panicking dude, foreigner not understanding anything, Japanese tourist, hero etc.

• Observers: played by project staff, they will act as „ghosts”, spectating, taking notes on evaluation charts, operating recording devices, but not interfering.

• Security: airport professionals led by the exercise security officer to secure test site and intervene in case any real danger will emerge.

• IT support: Zoltán Nagy (BZN) and BZN staff providing IT tools for data capture, process and analysis

Participating airport organizations

Airport Police Directorate

Part of HNP, performs general law enforcement plus aviation security tasks as well as operating CCTV surveillance. Evacuation decision can only be taken by the Directorate and it is the main responsible organization for executing emergency evacuation plans.

BUD-Sec Ltd.

Private security company of the airport operator, responsible for passenger and perimeter security.

BUD Airport Inc.

Operator of the airport, responsible for managent of terminal as a facility. For simplification purposes it will also play the role of ground handling organizations (operating check-in desks, information desk etc.)

Miscellaneous information and dress code

Those informations will be provided to the volunteers by BZN before the excercises.

• Participants are kindly advised to bring their cell phones, rest of equipment – if needed – it will be provided on the spot by the organizers.

• Clothing must be selected according to external weather conditions, as evacuation exits lead to open air. Rain is possible. Volunteers must have pens, and backpacks/handbags, if possible.

• Medical and security first responders will be present at test site.

Logistics

Transportation

• Participants are expected to arrive to T1 each exercise day by 08:00 CET, using public transport or by car.


page 79

• BUD T1 is located at the southwestern city limits of Budapest, it can be reached by train, bus or car during daytime. A parking place is available in front of the terminal. Parking will be provided free of charge for testers within the bounds of free spaces left.

Security check, briefing and informed consent

• For safety and security reasons, a quick security and safety check will be performed by the airport operator and supervised by the exercise safety officer shortly before the tests. All hold items will be examined and removed from sites, if needed. A short check of all CCTV and deployed cameras and screens/images is necessary before the exercises.

• Participants have to present their ID and sign the Informed Consent upon arrival. • Briefing for participants will commence at 09:00, in the following order:

o 09:00 to 09:20 – briefing Local Coordination Team and observers o 09:25 to 09:45 – briefing Test Support Unit

Utilities and catering

• The airport operator will take care for local utilities and ensure that all toilets, duty free areas and doors remain opened during the tests.

• Terminal 1 has a restaurant with catering possibilities, some toilets at landside-airside areas. The proximity of the building has roofed parts and dedicated area for smoking.

• For the exercises, about 2-300 students playing passengers will be at airside areas (with the possibility to use the Apron area), the rest will be deposed at the landside – departure and arrival parts, and external areas (roofed).

IT tools

• All participants will receive an RFID card and all doors - relevant for the exercise (approx.: 15 doors) - will be equipped with an RFID card reader. When participants pass thru doors, their identity (RFID card ID number) and the timestamp will be recorded by the card reader device.

• The internal airport CCTV system will be operated during the period of the exercise. The recording of the cameras will be analyzed off-line for people identification and counting.

• Additional high resolution (HD) cameras will be installed at the critical positions to improve the quality and the reliability of the CCTV system. The recording from the HD cameras will be processed off-line same way as the CCTV images.

• Mobile phone application will be deployed to the compatible mobile phone of the participants. The application can determine the indoor position of the phone using built-in sensors of the mobile device. The application will log the position and transfers it to a central server for further analysis.

• Battery powered Bluetooth beacons (iBeacons) will be deployed at the critical positions to improve indoor localization precision.

Evaluation and reporting tasks

General documentation

The Documentation Coordinator should document necessary information about the exercises and testing such as: scope, performance objectives, exercise instructions, exercise specifications, evaluations, significance of the role-playing and safety/security. Evaluators and observers shall follow the instructions of the Documentation Coordinator.

To obtain the aims of assessing the overall stress-test framework, some general documentations should be prepared, implemented and maintained:


page 80

No. Name of the document Actual status

1. Excercise plan

The excercise plan covers exercises for one occasion, or other specified planning period and is detailed. In the plan exercise aims are translated into exercises to ensure a coherent link. The plans for evaluation of exercises should be part of this plan and reflect the policy of continuous improvement.

This document itself will be used as excercise plan.

2. Exercise evaluation The quality of each exercise is evaluated so that the correct conclusions from the evaluation of the target group can be drawn and improvements concerning future exercises can be made. The Test Coordinator is responsible to evaluate the excercices. Check-lists could be use during and after the tests.

Check lists will be prepared.

3. Safety and security plan

To ensure the safety and security of all participants, staff and the test site, the exercise safety officer should develop an exercise safety and security plan. The Exercise Coordinator must approve the plan and any modifications should be discussed with the General Coordination Team. This plan should identify the site-specific hazards of each case study. The controls that will be used to control each of these specific hazards must be identified int he plan.

Existing local plans will be used during the tests. All necessary control of the mob will be guaranteed by trained police officers and emergency experts.

4. Records of all types

Written and digital records shall be made about all relevant information concerning the case studies. Basic info shall be recorded such as principally the time, place, event, action, decision, responsible and playing character, if needed. Impressions, opinions could be added to the reports. The Documentation Coordinator is responsible to handle all forms and check lists. The indicators examined will be also developed in a check list form to be assessed and evaluated in the meantime and after the case studies.

Report forms and check lists will be prepared.

5. DELTA Test report

The Test Coordinator will prepare the report within 1 month in the light of all relevant records, reports made in line with the case studies.

DELTA Test report will be prepared after the case studies.

Evaluation periods

• on-the-fly, in parallel with test events; • follow-up, after the test is finished; • using feedback of observers plus measurement results recorded by equipment (eg. time, video,

voice); • writing the DELTA Test Report (D5.6) after assessment of all evaluations.

On-the-fly evaluation

Observers deployed in the area are wearing visibility vests and have to be considered as „ghosts” by players, but evaded like physical obstacles (walking bastion). They have to be equiped with a proper device allowing


page 81

them to fill the evaluation chart (annexed) and a time measurement tool. Optionally they can be equipped with body or head-mounted cameras and microphones to record what is happening around them.

Observers in separated area (control room) are continously assessing the situation, taking notes and assumptions, feeding in test results and participate in the debriefing.

The Exercise Coordinator has the right to intervene at any time during the excercise. This absolute right comes from his general role and position. If the test is not following the test plan for any reason, appropriate steps has to be taken to re-establish normal test flow if possible without endangering participants. This can include for example:

• New events supporting the original storyline • Changing events to better support storyline • Re-scheduling of events • Repeating events • Announcing break • Halting the test • Removing persons from the test • Resuming the test • Cancelling the test

Follow-up evaluation

Follow-up evaluation consisting of:

• Gathering individual feedback of observers/documentation team and players; • Analysis of records made by on-the-fly observers, closely monitoring the test phases and

participants; • Gathering measures by sensors, mainly cameras and microphones recording the event as well as

time measurement; • Analyzing records of measures; • Consensus meeting for observers and (follow-up) evaluators to assess final conclusions; • Evaluation records; • Writing test report using evaluation records.


page 82

Response to internal review process

The Content of this Annex has been submitted as part of the periodic review report to the PO/EU/ Reviewers.


page 83

Resilience Joint Evaluation and Test Report (JET report)

Template for Resilience Joint Evaluation and Test Report

(JET report)

Report Title: Template for Resilience Joint Evaluation and Test Report (JET Report)

Author(s): A. Jovanovic, K. Tetlak, Z. Szekely, K. Øien, P. Auerkari, S. Eremic, J. Sanne, A. Reis

Responsible Project Partner:

BZN Contributing Project Partners:

EU-VRi, R-Tech, HNP, SINTEF, HNP, NIS

Document data:

File name / Release:

D5.2-Stress-test-framework_v41bc07032018 Release No.: 0.46

Pages: 84 No. of annexes: 5

Status: Draft Dissemination level: PU

Project title: SmartResilience: Smart Resilience Indicators for Smart Critical Infrastructures

Grant Agreement No.: 700621

Project No.: 12135

WP title: WP5: SCI-Resilience: Application in Smart City Case Studies

Deliverable No: D5.12

Date: Due date: April 30, 2018 Submission date: Not specified

Keywords: Stress-test, assessment, framework, case-studies

Reviewed by: Not foreseen Review date: n/a

Not foreseen Review date: n/a

Approved by Coordinator: A. Jovanović Approval date: March 6, 2018

Stuttgart, March 2018


page 84

List of Acronyms & Definitions

Acronyms

Acronym Definition BUD Budapest Airport

CI Critical Infrastructure

DCL Dynamic Checklist

EU European Union

FE Functional Exercise

FEMA Federal Emergency Management Agency

FL Functionality Level

FSE Full Scale Exercise

ISO International Organization for Standardization

KPI Key Performance Indicators

MCDM Multi-criteria Decision Making

MSEL Master Scenario Event List

NFPA National Fire Protection Association

RAE Resilience Assessment Exercise

RL Resilience Level

SCI Smart Critical Infrastructure

SimCel Simulation Cell

TTX Table-Top Exercise

VTT Technical Research Centre of Finland

Definitions

1. Absorption time It is defined as the time at which the SCI absorbs the disruptive event, decreasing its initial functionality. It is measured as the difference between t2 and t1 [8].

2. Alarm level Readiness increased further and/or some corrective actions triggered (e.g. RL=2.8).

3. Alert level Level at which the warning should be issued and the readiness level increased (e.g. RL=3.4).

4. Case Study A case study is used to analyze particular types of critical infrastructures. In the SmartResilience project 9 case studies are considered: ALPHA (financial system), BRAVO (smart city), CHARLIE (health care system), DELTA (airport), ECHO (refinery), FOXTROT (water supply), GOLF (transport), HOTEL (energy supply), and INDIA (cascading effects).

5. Critical level Level at which emergency measures are triggered and/or possible shut down of the infrastructure is required (e.g. RL=2.4).


page 85

6. Disruption time It is defined as the total time taken by the CI to recover. It is also seen as a measure for recover capacity of the SCI to return to a desired functionality level. In the FL-t curve, it is the time between when the event occurs i.e. at time t1 and time when the SCI has fully recovered i.e. t4 [8].

7. Downtime It is defined as time duration for which the functionality level of the infrastructure remains below the threshold level of the functionality. It is measured as the difference between t3 and t2 [8].

8. Drill [6] Coordinated, supervised activity usually employed to validate a specific function or capability in a single agency or organization. Drills are commonly used to provide training on new equipment, validate procedures, or practice and maintain current skills. For example, drills may be appropriate for establishing a community-designated disaster receiving center or shelter. Drills can also be used to determine if plans can be executed as designed, to assess whether more training is required, or to reinforce best practices. A drill is useful as a stand-alone tool, but a series of drills can be used to prepare several organizations to collaborate in an FSE. For every drill, clearly defined plans, procedures, and protocols need to be in place. Personnel need to be familiar with those plans and trained in the processes and procedures to be drilled. Drill [Task 5.2] [12] Activity which practices a particular skill often involves repeating the same thing several times. Drill (in-depth) [Task 5.2] [12] A drill is a coordinated, supervised activity usually employed to test a single specific operation or function in a single entity or multi-organization team.

9. Dynamic Checklist Dynamic checklist is a tool that helps to select the issues, elements and indicators for the resilience assessment or functionality assessment.

10. Element Most relevant function of a critical infrastructure that contributes to the overall functionality of that infrastructure.

11. Full scale exercise (FSE) FSEs are typically the most complex and resource-intensive type of exercise. They involve multiple agencies, organizations, and jurisdictions and validate many facets of preparedness. FSEs often include many players operating under cooperative systems such as the Incident Command System (ICS) or Unified Command. In an FSE, events are projected through an exercise scenario with event updates that drive activity at the operational level. FSEs are usually conducted in a real-time, stressful environment that is intended to mirror a real incident. Personnel and resources may be mobilized and deployed to the scene, where actions are performed as if a real incident had occurred. The FSE simulates reality by presenting complex and realistic problems that require critical thinking, rapid problem solving, and effective responses by trained personnel.

12. Functional Exercise (FE) Designed to validate and evaluate capabilities, multiple functions and/or sub-functions, or interdependent groups of functions. FEs are typically focused on exercising plans, policies, procedures, and staff members involved in management, direction, command, and control functions. In FEs, events are projected through an exercise scenario with event updates that drive activity typically at the management level. An FE is conducted in a realistic, real-time environment; however, movement of personnel and equipment is usually simulated. FE controllers typically use a Master Scenario Events List (MSEL) to ensure participant activity remains within predefined boundaries and ensure exercise objectives are accomplished. Simulators in a Simulation Cell (SimCell) can inject scenario elements to simulate real events.

13. Game [6] A simulation of operations that often involves two or more teams, usually in a competitive environment, using rules, data, and procedures designed to depict an actual or hypothetical situation.


page 86

Games explore the consequences of player decisions and actions. They are useful tools for validating plans and procedures or evaluating resource requirements. Game [Task 5.2] [12] A game is a simulation of operations that often involves two or more teams, usually in a competitive environment, using rules, data, and procedures designed to depict an actual or assumed real-life situation.

14. Improvement/adaptation/transformation It is the capacity of the SCI to learn from the disruptive event (e.g. a revision of plans, modification of procedures, introduction of new tools and technologies). It is measured as the ratio of change in functionality level after the event to normal operation [8].

15. Indicator In SmartResilience, indicator is the description of HOW to measure an issue. Any type/form of indicators are considered appropriate in the SmartResilience methodology, meaning that they can be yes/no questions, numbers, percentages, portions, or some other type. E.g., it can be "percentage of personnel in a certain response team taken a certain course" [8].

16. Issue Issue is a very general term referring to anything (factors, conditions, functions, actions, capacities, capabilities, etc.) that is important in order to be resilient against severe threats such as terror attacks, cyber threats and extreme weather. It is WHAT is important, and it is allocated to one of the five phases in the resilience cycle. E.g., it can be "training" performed in the anticipate/prepare phase [8].

17. Recovery rate is defined as the rate at which the SCI recovers from the disruptive event and gains its initial or desired functionality. It is measured as the ratio of change in functionality level between time t3 to t4 [8].

18. Recovery time It is defined as the time at which the SCI recovers from the disruptive event and gains its initial or desired functionality. It is measured as a difference between time t4 to t3 [8].

19. Resilience of an infrastructure is the ability to anticipate possible adverse scenarios/events (including the new/emerging ones) representing threats and leading to possible disruptions in operation/functionality of the infrastructure, prepare for them, withstand/absorb their impacts, recover optimally from disruptions caused by them and adapt to the changing conditions.

20. Resilience Assessment Exercise Resilience Assessment Exercise in this report covers any organized event in which the resilience is assessed, either in terms of Functionality Level Assessment (FL), Resilience Level assessment or other related resilience assessment analysis, e.g. using tools, checklists and similar, as well as stress-testing and resilience related decision-making. That includes the events such as “Seminar”, “Workshop”, “Table-Top Exercise”, “Game”, “Drill”, “Functional Exercise”, “Full scale exercise” (see: footnotes 13 to 19).

21. Robustness It is defined as the capacity of the SCI to endure the effects of negative event without significant deviation from the desired functionality level. It is measured as the ratio of the percentage of the lowest FL after the disruption i.e. at time, t2 to the FL at during normal operation i.e. at time, t0 [8].

22. Scenario We use the term "scenario" in two ways; one broad and one detailed way. In the broad way, we use it for a specific selection of critical infrastructures and threats for a given area/city, i.e. the selected area, critical infrastructures and threats. In the detailed way, e.g. for stress-testing, we define a very specific scenario describing a simulated extreme event [8]. Scenario [Task 5.2] [12] Pre-planned storyline that drives an exercise, the stimuli used to achieve exercise objectives.

23. Seminar Generally, a seminar orients participants to, or provide an overview of, authorities, strategies, plans,


page 87

policies, procedures, protocols, resources, concepts, and ideas. As a discussion-based exercise, seminars can be valuable for entities that are developing or making major changes to existing plans or procedures. Seminars can be similarly helpful when attempting to assess or gain awareness of the capabilities of interagency or inter-jurisdictional operations.

24. Smart Critical Infrastructure Based on the definition of smart systems from EPOSS and the work conducted in SmartResilience project T2.1 [9], smart critical infrastructure is a critical infrastructure whose functionality is enhanced by the use of smart (intelligent) systems and technologies.

25. Smartness The Smartness here implies that a critical infrastructure is becoming increasingly “smarter” (e.g. the “smart cities”). Making the infrastructures “smarter” usually means making them more adaptive, more intelligent in the normal operation and use. These aspects have been developed in [9].

26. Stress-test It is used to assess the safety margins of an infrastructure in case of a disruptive event, e.g., evaluating the infrastructure response when facing a set of adverse events (following the European Nuclear Safety Regulators Group stress-test definition) [11].

27. Table-Top Exercise (TTX) [6] A TTX is intended to generate discussion of various issues regarding a hypothetical, simulated emergency. TTXs can be used to enhance general awareness, validate plans and procedures, rehearse concepts, and/or assess the types of systems needed to guide the prevention of, protection from, mitigation of, response to, and recovery from a defined incident. Generally, TTXs are aimed at facilitating conceptual understanding, identifying strengths and areas for improvement, and/or achieving changes in perceptions. Tabletop Simulation (brief) [Task 5.2] [12] A tabletop exercise will include key personnel discussing simulated scenarios that involve disruptive event in an informal setting [around a table]. Tabletop exercises can be a tool to build competence and support for a revised plan or procedure; or, review plans, policies, and procedures or to assess the systems needed to respond to undesired situations. Participants are expected to discuss the issues that result from the simulated events and develop decisions through paced problem solving. Tabletop exercises can be timed with expected rapid decision making or untimed allowing for in depth discussion and development of solutions. Usually, untimed tabletop exercises will be used in this project.

28. Threat A threat is a low-probability, high consequence events, e.g. terror attacks, cyber-attacks and extreme weather. Such events – called "black swans" – are notoriously hard to plan for [8].

29. Workshop [6] Although similar to seminars, workshops differ in two important aspects: participant interaction is increased, and the focus is placed on achieving or building a product. Effective workshops entail the broadest attendance by relevant stakeholders. Products produced from a workshop can include new standard operating procedures (SOPs), emergency operation plans, continuity of operations plans, or mutual aid agreements. To be effective, workshops should have clearly defined objectives, products, or goals, and should focus on a specific issue. Workshop (brief) [Task 5.2] [12] Workshops are designed to orient participants to new or updated plans, policies, or procedures. They are unconstrained by real-time simulation of events and are facilitated by an experienced presenter. Participant interaction is required, and the focus is on achieving a result. In the current context, it means invited experts of a critical infrastructure will analyze a given scenario in view of selected issues and indicators, providing sample data for the indicators where possible.


page 88

Summary / Main facts & findings

Please SHORTLY describe here*: • who did the assessment (e.g. Resilience Exercise Manager, Main Analyst) • the type of critical infrastructure assessed • the threat(s) against which the assessment was performed • the type of exercise (e.g. seminar, workshop, table-top exercise, game, drill,

functional exercise, full-scale exercise) • type of scenario(s) taken into account in case of a drill or a tabletop • the main result of the exercise • other significant information that was collected • recommendations to the management

Example**:

The Hungarian National Police organized a large-scale drill on Budapest Airport T1, involving more than 1000 people, including non-professionals as actors. The drill was focused on combinations of cyberattack, bombing attempt and disastrous industrial accident. Evacuations in different directions (two-direction, seal-off in the terminal and one direction evacuation after) has been exercised in three phases for a whole day. Main result is that the smart devices of the passengers can be used for smart localization and controlled evacuation and this makes evacuations faster and more secure. Other significant information is that the terminal is back to operation faster if we push information on changed flight details to smart devices of the passengers first and allow them to re-enter the terminal after. Indoor localization also showed that there is a bottleneck at the line of passenger security control, even if the control itself is suspended. It has been turned out that in case a cyber-attack neutralizes the fire alarm system, the evacuation message is not playable from the terminal’s audio system. Main recommendations to increase resilience:

- in case only the terminal is not operational, allow access for passengers evacuated from the airside after checking in, to other boarding facilities;

- auxiliary tools (e.g. loudspeaker) are needed to spread evacuation messages - cell and/or wireless coverage is one of the most vital part of all operations,

including evacuation, therefore it has to be kept operational as long as possible.

* - All recommendations for the contents are provided as the text style “Hidden” (yellow text)

** - The recommendations are sometimes followed by an “Example” (the examples are printed in Times New Roman italic)


page 89

Introduction

This text should possibly remain the same for all case studies.

The textual part describes in a short and plain manner what is the current resilience value of the given critical infrastructure and how it is possible to improve it in response to the threat scenarios described in this report. The assessment part provides information about the assessments or tests carried out following the SmartResilience methodologies. The assessment part also allows cross-sectoral comparison and benchmarking.

NOTE: For different types of Resilience Assessment Exercise see: “Definitions”.


page 90

Assessed Critical infrastructure

2.1 General description of the assessed critical infrastructure Please provide a half page description including a picture. Describe the infrastructure, in which sector it is,

what part or facility will be assessed (if not the whole). Describe key performance indicators (KPIs) justifying why it is a critical infrastructure, for example economic significance, service coverage, national/regional/global importance, production volume, etc. Alternatively, in the cases where the use of KPI is not possible a qualitative explanation may suffice.

2.2 Smart critical infrastructure features Please provide up to one page. Describe how does smart technology feature in the assessment of critical

infrastructure. Present sector-specific smart maturity levels if there are any. Compare this with the smartness maturity model developed in the SmartResilience project to assess what would be the smartness level of the critical infrastructure. Provide information on ongoing or planned developments towards «smarter» critical infrastructure, especially if the threat is related to it.


page 91

Assessment Setup

3.1 Threat Please provide no more than half page explanation. Describe what kind of threat(s) was covered in this

assessment. Explain why this threat was selected. Please emphasize how the threat is or can be linked to the « smartness » of the infrastructure in terms of integration and interconnectedness, intelligence, and autonomy.

3.2 Scenario Please provide maximum one and a half page description. For a definition of scenario see: “Definitions”.

Describe it shortly like a movie resume. Use tables, pictures or figures to present the flow of events. NOTE: Whereas threat is general, scenario is a specific description.

3.3 Issues/elements/indicators refinement These refinements can follow the recommendation of the report [10].

3.3.1 Selection of issues/elements/indicators Provide a 5-6 lines description about the most relevant comments which have been made in regards to the

selected issues/elements/indicators.

3.3.2 Quality assurance Please describe gaps and general quality matters relevant for the scenario. Provide the gaps and their

improvement from a usefulness and completeness perspective


page 92

Description of the exercise method (type of event) and other practical details

4.1 Exercise method Please provide maximum one and a half page description. Define the approach used. Justify it. Describe

when, where and with whom was the assessment carried out. Reflect on organizational, logistical, safety and security aspects where applicable (drill, exercise). If it includes recording, tracking or identifying people in other means, present the privacy measures taken. Explain how the method is related to threat(s), scenario(s) and especially smart critical infrastructure features, for example how to capture data or which external data was used during the assessment.

4.2 Stakeholders involved in the exercise Please, provide only 2-3 lines explanation. Please list the who are the stakeholders, the type of

organization, and the criteria used to select them. Additional comments can also be provided (e.g. “bank representative missing”).

4.3 Planning of the exercise Please, provide only 2-3 lines explanation that describe how the activity is planned.

4.4 Informed consent Please provide a brief description (4-5 lines) of who and when signed the informed consent. Please ensure,

that every workshop participant signs the informed consent when formally requested (e.g. in EU-funded projects). The template for the informed consent (see [10]) must be collected and stored by the Resilience Assessment Manager for future references.


page 93

Results

Perform the assessment of the smart critical infrastructure in the sub-sections below.

5.1 Main results Please provide half page description. Draw one main conclusion regarding resilience of the selected smart

critical infrastructure. Use plain language.

5.2 Other information Please provide one page description. Here other results of the assessment can be included, mainly

connected to the Dynamic Checklists (DCLs). Please ensure that it is easy to read. Examples of the “results of the assessment” can be, e.g., how are the issues/elements/indicators useful for the assessments, which ones can be used for comparison with other infrastructures, and what other issues/elements/indicators can be considered for resilience improvement.


page 94

Recommendations

Please provide half page description. Provide recommendations for the management of the assessed critical infrastructure. Clear statements on needs.

Please suggest main improvements for current practices of resilience assessment management (e.g. Business Continuity Planning, Risk and Vulnerability Analysis, Crisis and Emergency Planning, etc.)


page 95

Appendix 1 Resilience Assessment Report Guideline

GENERAL

This section provides an explanation how to fill-in the Resilience Assessment Report correctly. The following template has been prepared based on the Federal Emergency Management Agency Action Request Form.

Envisaged time for filling in this form in the SmartResilience Tool: 30 minutes.

Nevertheless, time for reviewing instructions, searching existing data sources, gathering and maintaining the needed data (from SmartResilience database) may vary from case to case. This form can be accessed online via: http://www.smartresilience2.eu-vri.eu/RunningApp/RIdb/Report.aspx

INSTRUCTIONS

Items on the Resilience Assessment Report that are not specifically listed below are considered self-explanatory.

{1} The Resilience Assessment Exercise follows the structure shown in Figure 1 which is adapted from the National Fire Protection Association [4] and from D5.2 in the SmartResilience project [5].

Team Members

IT/ SCADA/ data specialist*

Infrastructure Specialist(s)*

Other experts / specialist, (e.g.

PR)*

Resilience Exercise Manager

Executive Team

Liaison Officer/ Security Liaison Officer*

Resilience Tools operator(s)*

Main Analyst

Safety & Security/ Rescue Services

* - optional

Requestor*

Figure 1: Resilience Assessment Exercise structure adapted from [4] and [5]

{2} The Requestor can be the European Commission (EC)/local authorities/company authorities/individual person. {3} The Executive Team should be understood as FEMA's "Command Staff" as on Figure 1: staff who report directly to the Resilience Exercise Manager (this corresponds to FEMA’s Incident Commander), including the

http://www.smartresilience2.eu-vri.eu/RunningApp/RIdb/Report.aspx


page 96

Main Analyst, the Liaison Officer/Security Liaison Officer, Resilience Tools Operator(s). They may have an assistant or assistants, as needed [7]. {4} The Liaison Officer/Security Liaison Officer is a member of the Executive Team (this corresponds to FEMA’s Command Staff) responsible for coordinating with representatives from cooperating and assisting agencies [7]. {5} The Resilience Tools Operator is the person responsible for putting the data (indicators, elements, etc.) into the database and conducting the assessment. {6} In case there are a large number of team members, please list key personnel only. {7} The hierarchy of the resilience assessments, as structured in the SmartResilience tool, is presented in Figure 2.

CASE STUDY #1

Scenario #1

Dynamic CheckList #1

Assement #1

Scenario #2...

...

...

Figure 2: Hierarchy of the resilience assessment(s) as structured in the SmartResilience tool

{8} For the scenario description, please describe the scenario(s) that is analyzed during the exercise. Multiple scenarios can be described and applied for different critical infrastructures and threats. {9} For particular substructures (parts of the infrastructure) involved in the exercise, please shortly explain what are the substructures of the given critical infrastructure involved in the case study. {10} Regarding the smartness level of the critical infrastructure, please shortly explain how the infrastructure is characterized in terms of integration and interconnectedness, intelligence, and autonomy and assess its smartness maturity level (vide: SmartResilience D.2.1: Understanding "smart technologies and their role in ensuring resilience of infrastructures [9]). {11} In case of a drill or a table-top exercise, please describe the logistical preparations and support needed for the event. In case of a workshop, only fill if there was a field trip, an interpreter (for translation), etc., which is important for the results analysis. {12} The case study “identifier” and name should only be filled for SmartResilience case studies (to be completed by Resilience Assessment Manager). {13} In order to choose the type of event (cf. FEMA 2003 [6]), please consider the following definitions:

Seminar Generally, a seminar orients participants to, or provide an overview of, authorities, strategies, plans, policies, procedures, protocols, resources, concepts, and ideas. As a discussion-based exercise, seminars can be valuable for entities that are developing or making major changes to existing plans or procedures. Seminars can be similarly helpful when attempting to assess or gain awareness of the capabilities of interagency or inter-jurisdictional operations. Workshop [6] Although similar to seminars, workshops differ in two important aspects: participant interaction is increased, and the focus is placed on achieving or building a product. Effective workshops entail the broadest attendance by relevant stakeholders. Products produced from a workshop can include new standard operating procedures (SOPs), emergency operation plans, continuity of operations plans, or mutual aid agreements. To be effective, workshops should have clearly defined objectives, products, or goals, and should focus on a specific issue. Table-Top Exercise (TTX) [6] A TTX is intended to generate discussion of various issues regarding a hypothetical, simulated emergency.


page 97

TTXs can be used to enhance general awareness, validate plans and procedures, rehearse concepts, and/or assess the types of systems needed to guide the prevention of, protection from, mitigation of, response to, and recovery from a defined incident. Generally, TTXs are aimed at facilitating conceptual understanding, identifying strengths and areas for improvement, and/or achieving changes in perceptions. Game [6] A simulation of operations that often involves two or more teams, usually in a competitive environment, using rules, data, and procedures designed to depict an actual or hypothetical situation. Games explore the consequences of player decisions and actions. They are useful tools for validating plans and procedures or evaluating resource requirements. Drill [6] Coordinated, supervised activity usually employed to validate a specific function or capability in a single agency or organization. Drills are commonly used to provide training on new equipment, validate procedures, or practice and maintain current skills. For example, drills may be appropriate for establishing a community-designated disaster receiving center or shelter. Drills can also be used to determine if plans can be executed as designed, to assess whether more training is required, or to reinforce best practices. A drill is useful as a stand-alone tool, but a series of drills can be used to prepare several organizations to collaborate in an FSE. For every drill, clearly defined plans, procedures, and protocols need to be in place. Personnel need to be familiar with those plans and trained in the processes and procedures to be drilled. Functional Exercise (FE) Designed to validate and evaluate capabilities, multiple functions and/or sub-functions, or interdependent groups of functions. FEs are typically focused on exercising plans, policies, procedures, and staff members involved in management, direction, command, and control functions. In FEs, events are projected through an exercise scenario with event updates that drive activity typically at the management level. An FE is conducted in a realistic, real-time environment; however, movement of personnel and equipment is usually simulated. FE controllers typically use a Master Scenario Events List (MSEL) to ensure participant activity remains within predefined boundaries and ensure exercise objectives are accomplished. Simulators in a Simulation Cell (SimCell) can inject scenario elements to simulate real events. Full scale exercise (FSE) FSEs are typically the most complex and resource-intensive type of exercise. They involve multiple agencies, organizations, and jurisdictions and validate many facets of preparedness. FSEs often include many players operating under cooperative systems such as the Incident Command System (ICS) or Unified Command. In an FSE, events are projected through an exercise scenario with event updates that drive activity at the operational level. FSEs are usually conducted in a real-time, stressful environment that is intended to mirror a real incident. Personnel and resources may be mobilized and deployed to the scene, where actions are performed as if a real incident had occurred. The FSE simulates reality by presenting complex and realistic problems that require critical thinking, rapid problem solving, and effective responses by trained personnel.

{14} For other details regarding the Exercise, please indicate any additional details about the exercise, e.g. incidents that occurred during the exercise, number of signed Informed Consents, etc. {15} For other details regarding the SmartResilience assessment analysis setup, please insert the timetable and the workflow. In case of a drill or a table-top, please indicate also fictitious (gameplay) timespan and locations (scenes). Provide pictures/figures whenever available. {16} The Dynamic Checklist ID is derived out of the SmartResilience tool (automatic unique number). {17} Alert level Level at which the warning should be issued and the readiness level increased (e.g. RL=3.4). {18} Alarm level Readiness increased further and/or some corrective actions triggered (e.g. RL=2.8).


page 98

{19} Critical level Level at which emergency measures are triggered and/or possible shut down of the infrastructure is required (e.g. RL=2.4). {20} Stress-test It is used to assess the safety margins of an infrastructure in case of a disruptive event, e.g., evaluating the infrastructure response when facing a set of adverse events (following the European Nuclear Safety Regulators Group stress-test definition) [11]. {21} The graphical representation of the functionality parameters are presented in Figure 3.

Event

100%

Func

tiona

lity

Leve

l (FL

)

SCENARIO time

t0 t1

t3

t5'

t2

t4

Under-stand Risk

Respond/Recover

Anti-cipate/Prepare

Loss of functionality

Improvement/adaptation/

transformation

Down-time

Disruption time

Robustness

t5

Absorb/Withstand Adapt/ Transform

FL

FL’

Recovery time

Absorption time

Figure 3: Functionality parameters [8]

{22} Robustness It is defined as the capacity of the SCI to endure the effects of negative event without significant deviation from the desired functionality level. It is measured as the ratio of the percentage of the lowest FL after the disruption i.e. at time, t2 to the FL at during normal operation i.e. at time, t0 [8]. {23} Disruption time It is defined as the total time taken by the CI to recover. It is also seen as a measure for recover capacity of the SCI to return to a desired functionality level. In the FL-t curve, it is the time between when the event occurs i.e. at time t1 and time when the SCI has fully recovered i.e. t4 [8]. {24} Absorption time It is defined as the time at which the SCI absorbs the disruptive event, decreasing its initial functionality. It is measured as the difference between t2 and t1 [8]. {25} Downtime It is defined as time duration for which the functionality level of the infrastructure remains below the threshold level of the functionality. It is measured as the difference between t3 and t2 [8].


page 99

{26} Recovery time It is defined as the time at which the SCI recovers from the disruptive event and gains its initial or desired functionality. It is measured as a difference between time t4 to t3 [8]. {27} Recovery rate is defined as the rate at which the SCI recovers from the disruptive event and gains its initial or desired functionality. It is measured as the ratio of change in functionality level between time t3 to t4 [8]. {28} Improvement/adaptation/transformation It is the capacity of the SCI to learn from the disruptive event (e.g. a revision of plans, modification of procedures, introduction of new tools and technologies). It is measured as the ratio of change in functionality level after the event to normal operation [8]. {29} The approval of the resilience assessment results mean that the Report has been created as a result of the event (exercise), not as a pre-existing condition. {30} For the comparison of the results obtained by the DCL-based assessment with the results obtained through other resilience and risk assessment methods, consider e.g. ISO 22316:2017 [3]. In case that additional space is needed or to provide more information, please indicate "see attached" (and list them in section VIII. 12/ IX. 12).


page 100

Appendix 2 Resilience Assessment Report Template


page 101

RESILIENCE ASSESSMENT REPORT

The template is proposed in the EU funded project: SmartResilience (the Grant Agreement No. 700621) http://smartresilience.eu-vri.eu/

NOTE: All the fields should be filled out. For the data which are, e.g.: not applicable, not known, not available, etc., the respective entry should be inserted.

Infrastructure assessed:

Scenario name & ID:

DCL name & ID:

Assessment name & ID:

Date:

Executive summary of the exercise: Historical data/ situational reporting of similar events (real or simulated):

Main objectives and challenges of the exercise:

Description of the conducted exercise:

Main findings after the exercise:

Can be copied from section: 5 Results

http://smartresilience.eu-vri.eu/


page 102

Part A: Basic info I. Resilience assessment/stress-test team1 member’s information: Requestor

Requestor’s initials & last name2: Requestor’s organization: Requestor’s position:

Requestor’s phone number: Requestor’s email address:

II. Resilience assessment/stress-test team member’s information: Resilience Assessment Exercise (RAE) Manager

RAE Manager’s initials / last name: RAE Manager’s organization: RAE Manager’s position:

RAE Manager’s phone number:

RAE Manager’s email address:

III. Resilience assessment/stress-test team member’s information: Executive Team3

Main Analyst’s initials / last name:

Main Analyst’s organization:

Main Analyst’s position:

Liaison Officer4/Security Liaison Officer’s initials / last name (if applicable):

Liaison Officer/ Security Liaison Officer’s organization (if applicable):

Liaison Officer/ Security Liaison Officer’s position (if applicable):

Resilience Tool Operator’s initials / last name (if applicable)5:

Resilience Tool Operator’s organization (if applicable):

Resilience Tool Operator’s position (if applicable):

IV. Resilience assessment/stress-test member’s information: Team Members6

Infrastructure Specialist’s initials / last name (if applicable):

1.

2.

Infrastructure Specialist’s organization (if applicable):

Infrastructure Specialist’s position (if applicable):

Other Experts’ initials / last name (if applicable):

1.

2.

Safety & Security/ Rescue Specialists’ initials / last name (if applicable):

IT/SCADA/data specialists’ initials / last name (if applicable):

V. Scenario information7 (to be completed by the Resilience Assessment Exercise Manager) Scenario name:

Scenario description8:

1 The Resilience Assessment Exercise follows the structure shown in Figure 1 which is adapted from the National Fire

Protection Association [4] and from D5.2 in the SmartResilience project [5]. 2 The Requestor can be the European Commission (EC)/local authorities/company authorities/individual person. 3 The Executive Team should be understood as FEMA's "Command Staff" as on Figure 1: staff who report directly to the

Resilience Exercise Manager (this corresponds to FEMA’s Incident Commander), including the Main Analyst, the Liaison Officer/Security Liaison Officer, Resilience Tools Operator(s). They may have an assistant or assistants, as needed [7].

4 The Liaison Officer/Security Liaison Officer is a member of the Executive Team (this corresponds to FEMA’s Command Staff) responsible for coordinating with representatives from cooperating and assisting agencies [7].

5 The Resilience Tools Operator is the person responsible for putting the data (indicators, elements, etc.) into the database and conducting the assessment.

6 In case there are a large number of team members, please list key personnel only. 7 The hierarchy of the resilience assessments, as structured in the SmartResilience tool, is presented in Figure 2. 8 For the scenario description, please describe the scenario(s) that is analyzed during the exercise. Multiple scenarios can

be described and applied for different critical infrastructures and threats.


page 103

Type(s) of (smart) critical infrastructure involved:

- Financial Systems

- Transportation System

- ICT Systems

- Energy Supply System

- Industrial Production Systems

- Health Care System

- Water Supply System

- Other SCIs (describe)

Particular substructures (parts of the infrastructure) involved in the exercise9:

Provide details on the smartness level of the selected infrastructure10:

Other (description/details, e.g. logistical, organizational)11:

Other CI(s) possibly affected:

Type(s) of threats: - All/any threats

- Natural threats

- Cascading effects

- Terrorist attack

- Social unrest

- Cyber attack

- New technology accident

- Other threats (describe)

Other (description/details):

Task Nr.12:

Case Study “identifier” and nameError! Bookmark not defined.:

VI. EXCERCISE INFORMATION (to be completed by Resilience Assessment Exercise Manager)

Start date, time:

End date, time:

Event place/venue:

9 For particular substructures (parts of the infrastructure) involved in the exercise, please shortly explain what are the

substructures of the given critical infrastructure involved in the case study. 10 Regarding the smartness level of the critical infrastructure, please shortly explain how the infrastructure is

characterized in terms of integration and interconnectedness, intelligence, and autonomy and assess its smartness maturity level (vide: SmartResilience D.2.1: Understanding "smart technologies and their role in ensuring resilience of infrastructures [9]).

11 In case of a drill or a table-top exercise, please describe the logistical preparations and support needed for the event. In case of a workshop, only fill if there was a field trip, an interpreter (for translation), etc., which is important for the results analysis.

12 The case study “identifier” and name should only be filled for SmartResilience case studies (to be completed by Resilience Assessment Manager).


page 104

Type event (cf. FEMA 2013 [6]): - Seminar13

- Workshop14

- Table-Top Exercise15

- Game16

- Drill17

- Functional Exercise18

- Full-Scale Exercise19

- Other (describe)

13 Seminar

Generally, a seminar orients participants to, or provide an overview of, authorities, strategies, plans, policies, procedures, protocols, resources, concepts, and ideas. As a discussion-based exercise, seminars can be valuable for entities that are developing or making major changes to existing plans or procedures. Seminars can be similarly helpful when attempting to assess or gain awareness of the capabilities of interagency or inter-jurisdictional operations.

14 Workshop [6] Although similar to seminars, workshops differ in two important aspects: participant interaction is increased, and the focus is placed on achieving or building a product. Effective workshops entail the broadest attendance by relevant stakeholders. Products produced from a workshop can include new standard operating procedures (SOPs), emergency operation plans, continuity of operations plans, or mutual aid agreements. To be effective, workshops should have clearly defined objectives, products, or goals, and should focus on a specific issue.

15 Table-Top Exercise (TTX) [6] A TTX is intended to generate discussion of various issues regarding a hypothetical, simulated emergency. TTXs can be used to enhance general awareness, validate plans and procedures, rehearse concepts, and/or assess the types of systems needed to guide the prevention of, protection from, mitigation of, response to, and recovery from a defined incident. Generally, TTXs are aimed at facilitating conceptual understanding, identifying strengths and areas for improvement, and/or achieving changes in perceptions.

16 Game [6] A simulation of operations that often involves two or more teams, usually in a competitive environment, using rules, data, and procedures designed to depict an actual or hypothetical situation. Games explore the consequences of player decisions and actions. They are useful tools for validating plans and procedures or evaluating resource requirements.

17 Drill [6] Coordinated, supervised activity usually employed to validate a specific function or capability in a single agency or organization. Drills are commonly used to provide training on new equipment, validate procedures, or practice and maintain current skills. For example, drills may be appropriate for establishing a community-designated disaster receiving center or shelter. Drills can also be used to determine if plans can be executed as designed, to assess whether more training is required, or to reinforce best practices. A drill is useful as a stand-alone tool, but a series of drills can be used to prepare several organizations to collaborate in an FSE. For every drill, clearly defined plans, procedures, and protocols need to be in place. Personnel need to be familiar with those plans and trained in the processes and procedures to be drilled.

18 Functional Exercise (FE) Designed to validate and evaluate capabilities, multiple functions and/or sub-functions, or interdependent groups of functions. FEs are typically focused on exercising plans, policies, procedures, and staff members involved in management, direction, command, and control functions. In FEs, events are projected through an exercise scenario with event updates that drive activity typically at the management level. An FE is conducted in a realistic, real-time environment; however, movement of personnel and equipment is usually simulated. FE controllers typically use a Master Scenario Events List (MSEL) to ensure participant activity remains within predefined boundaries and ensure exercise objectives are accomplished. Simulators in a Simulation Cell (SimCell) can inject scenario elements to simulate real events.


page 105

Other (description/details)20:

19 Full scale exercise (FSE)

FSEs are typically the most complex and resource-intensive type of exercise. They involve multiple agencies, organizations, and jurisdictions and validate many facets of preparedness. FSEs often include many players operating under cooperative systems such as the Incident Command System (ICS) or Unified Command. In an FSE, events are projected through an exercise scenario with event updates that drive activity at the operational level. FSEs are usually conducted in a real-time, stressful environment that is intended to mirror a real incident. Personnel and resources may be mobilized and deployed to the scene, where actions are performed as if a real incident had occurred. The FSE simulates reality by presenting complex and realistic problems that require critical thinking, rapid problem solving, and effective responses by trained personnel.

20 For other details regarding the Exercise, please indicate any additional details about the exercise, e.g. incidents that occurred during the exercise, number of signed Informed Consents, etc.


page 106

Part B: Resilience Assessment Setup VII. SmartResilience analysis setup (to be completed by the Resilience Exercise Manager)

Type of resilience analysis: - resilience level assessment (RL)

- stress-test / functionality assessment (FL)

- other (describe)

Other (description/ details)21:

Dynamic Check-List (DCL) ID22: DCL name:

Issues and indicators for Resilience Level assessment (RL)* with their IDs: * - alternatively attach the full list as Appendix

Resilience Level (RL) zones: Alert level23:

Alarm level24:

Critical level25:

Elements and indicators for Functionality Level assessment (FL)* with their IDs: * - alternatively attach the full list as Appendix

: Functionality parameters26: Robustness (%)27:

Disruption time (minutes, days, etc.)28:

21 For other details regarding the SmartResilience assessment analysis setup, please insert the timetable and the

workflow. In case of a drill or a table-top, please indicate also fictitious (gameplay) timespan and locations (scenes). Provide pictures/figures whenever available.

22 The Dynamic Checklist ID is derived out of the SmartResilience tool (automatic unique number). 23 Alert level

Level at which the warning should be issued and the readiness level increased (e.g. RL=3.4). 24 Alarm level

Readiness increased further and/or some corrective actions triggered (e.g. RL=2.8). 25 Critical level

Level at which emergency measures are triggered and/or possible shut down of the infrastructure is required (e.g. RL=2.4).

26 The graphical representation of the functionality parameters are presented in Figure 3. 27 Robustness

It is defined as the capacity of the SCI to endure the effects of negative event without significant deviation from the desired functionality level. It is measured as the ratio of the percentage of the lowest FL after the disruption i.e. at time, t2 to the FL at during normal operation i.e. at time, t0 [8].

28 Disruption time It is defined as the total time taken by the CI to recover. It is also seen as a measure for recover capacity of the SCI to return to a desired functionality level. In the FL-t curve, it is the time between when the event occurs i.e. at time t1 and time when the SCI has fully recovered i.e. t4 [8].


page 107

Absorption time (minutes, days, etc.)29:

Downtime (minutes, days, etc.)30:

Recovery time (minutes, days, etc.)31:

Recovery rate (% over time)32:

Improvement/adaptation/transformation (%)33:

Part C: Resilience Assessment Results VIII. Resilience level assessment results

Resilience level assessment performance date:

Location:

Resilience Level assessment: * - alternatively attach the full list as Appendix x

Evaluation of Resilience Level assessment*: * - alternatively attach as Appendix

Evaluation of the results compared to the critical levels:

Resilience Level:

Critical levels to which belongs the results:

Preventative/ protective/ corrective measures to be implemented:

MCDM results: * - if applicable

Selected alternative:

Other relevant information:

29 Absorption time

It is defined as the time at which the SCI absorbs the disruptive event, decreasing its initial functionality. It is measured as the difference between t2 and t1 [8].

30 Downtime It is defined as time duration for which the functionality level of the infrastructure remains below the threshold level of the functionality. It is measured as the difference between t3 and t2 [8].

31 Recovery time It is defined as the time at which the SCI recovers from the disruptive event and gains its initial or desired functionality. It is measured as a difference between time t4 to t3 [8].

32 Recovery rate is defined as the rate at which the SCI recovers from the disruptive event and gains its initial or desired functionality. It is measured as the ratio of change in functionality level between time t3 to t4 [8].

33 Improvement/adaptation/transformation It is the capacity of the SCI to learn from the disruptive event (e.g. a revision of plans, modification of procedures, introduction of new tools and technologies). It is measured as the ratio of change in functionality level after the event to normal operation [8].


page 108

Approved by (name, affiliation)34 Date:

List of attachments:

IX. Functionality level assessment/stress-test results

Resilience level assessment/stress-test performance date:

Location:

Functionality Level assessment / stress-test results: * - alternatively attach the full list as Appendix

Evaluation of Functionality Level assessment /stress-test results: * - alternatively attach the full list as Appendix

Assessment of the functionality results compared to minimum / critical level of functionality / stress-test limits:

Robustness (%):

Is it equal/ above threshold:

Disruption time (minutes, days, etc.):


Absorption time (minutes, days, etc.):


Downtime (minutes, days, etc.):


Recovery time (minutes, days, etc.):


Recovery rate (% over time):


Improvement/adaptation/transformation (%):


Preventative/ protective/ corrective measures to be implemented:

MCDM results:

Selected alternative:

Other relevant information:

Approved by (name, affiliation): Date:

List of attachments:

34 The approval of the resilience assessment results mean that the Report has been created as a result of the event

(exercise), not as a pre-existing condition.


page 109

X. Feedback from the resilience assessment exercise Issues/ suggestions for the methodologies:

Issues / suggestions for the tools:

Comparison of the results obtained by the DCL-based assessment with the results obtained by other resilience or risk assessment methods35:

New indicators which have been derived from the dataset:

Other suggestions/general feedback:

35 For the comparison of the results obtained by the DCL-based assessment with the results obtained through other

resilience and risk assessment methods, consider e.g. ISO 22316:2017 [3].


page 110

References for Annex 4

[1] International organization for Standardization (2012). ISO Standard 22300:2012 Societal security -- Terminology, Geneva.

[2] International organization for Standardization (2012). ISO Standard 22398:2013 Societal security – Guidelines for exercises, Geneva.

[3] International organization for Standardization (2017). ISO Standard 22316:2017 Security and resilience -- Organizational resilience -- Principles and attributes, Geneva

[4] National Fire Protection Association, NFPA 1600 Standard on Disaster/Emergency Management and Business Continuity / Continuity of Operations Programs, Quincy, Massachusetts

[5] SmartResilience (2017). Deliverable D5.2: Stress-test and evaluation framework, EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany

[6] US Department for Homeland Security (2013), Homeland Security Exercise and Evaluation Program (HSEEP), https://www.fema.gov/media-library-data/20130726-1914-25045-8890/hseep_apr13_.pdf accessed on: December 20, 2017.

[7] Federal Emergency Management Agency; Glossary, https://training.fema.gov/emiweb/is/icsresource/assets/icsglossary.pdf accessed on: January 16, 2018

[8] SmartResilience (2017). Deliverable D3.6: Guideline for assessing, predicting and monitoring resilience of SCIs. EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany. [In progress]

[9] SmartResilience (2017). Deliverable D2.1: Understanding “smart” technologies and their role in ensuring resilience of infrastructure. EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany

[10] SmartResilience (2017). Deliverable D4.4: Common Protocol – A uniform process for continuous quality control. EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany

[11] EU (2012). European Nuclear Safety Regulators Group, https://ec.europa.eu/energy/sites/ener/files/documents/EC%20ENSREG%20Joint%20Statement%2026%20April%202012%20-Final%20to%20publish.pdf, accessed on: February 19, 2018

[12] SmartResilience (2017). Deliverable D5.2: Stress-test and evaluation framework. EU project SmartResilience, Project No. 700621 (2016-2019), Contact: EU-VRi, Stuttgart, Germany.

https://www.fema.gov/media-library-data/20130726-1914-25045-8890/hseep_apr13_.pdf

https://training.fema.gov/emiweb/is/icsresource/assets/icsglossary.pdf

https://ec.europa.eu/energy/sites/ener/files/documents/EC%20ENSREG%20Joint%20Statement%2026%20April%202012%20-Final%20to%20publish.pdf

https://ec.europa.eu/energy/sites/ener/files/documents/EC%20ENSREG%20Joint%20Statement%2026%20April%202012%20-Final%20to%20publish.pdf

Documents

SmartResilience - D5.2: Stress-test framework · 2018-03-23 · specific stresstest scenarios, which will be carried out as an event (- workshop, table top simulation, game, or drill),