Deliverable 13.12

The SIAM User Forum

Report

D r . L e o n H e m p e l

L a r s O s t e r m e i e r

T o b i a s S c h a a f

T e c h n i c a l U n i v e r s i t y B e r l i n

SIAM Security Impact Assessment

Measures

User forum report

Project number

261826

Call (part) identifier

FP7-Security-2010-1

Funding scheme Collaborative Project

1

TABLE OF CONTENTS

1. Introduction

1. Objectives 2

2. User forum outline 2

3. Summary of results 3

Feedback on: General 3

Feedback on: Functionalities of the AST 4

Feedback on: SMT Typology 5

Feedback on: Definitions of Roles 5

Feedback on: Questions 5

Feedback on: Reporting 6

4. Conclusion 7

5. Appendix 8

2

INTRODUCTION

1. OBJECTIVES

This report summarizes the results of the second user forum, conducted in Berlin on

October 31st 2013. After consulting the European Commission and leading

practitioners in the field of mass transportation, the SIAM consortium decided to

combine the four user forums into one international forum to enable in-depth

discussion within a heterogeneous group of international participants about the

assessment methodology, tools, and content developed by the SIAM consortium.

Presented was an alpha-version of the ‘SIAM Assessment Support Toolkit’ (SIAM

AST) which is based on the ‘SIAM Wireframe’ presented in the first round of user

fora in December 2012 and January 2013. The outcomes of these fora (D13.11)

enabled the adjustment of requirements by the potential users and the creation of

the SIAM AST. The toolkit, although in a stage of development, allowed the users to

interact with and inspect all phases and steps of the assessment support process and

simulate a full assessment. Any feature not fully available yet was indicated

sufficiently such that users could understand what this feature was intended to do.

In contrast to the first wireframe, it was now possible to present to the participants a

comprehensive set of assessment questions which the SIAM partners have been

developing within the course of the year. The objectives of the forum were to elicit

additional feedback on the revised toolkit features and to evaluate the assessment

questions.

2. User forum outline

The user forum was conducted at the “PC College” training facility in Berlin,

Germany. The facility had provided a local area network within which the SIAM AST

infrastructure could be simulated, by means of a local web application server

(provided by KU).

3

After a short introduction by the coordinator, the AST toolkit was presented by

Ronald Grau of Kingston University, covering a brief overview on the general ideas,

system architecture, and some of the important theoretical underpinnings of the

assessment support functionality.

Each participant was provided with a user account for the system and assigned a

specific role within the AST, fitting their professional expertise. It was then presented

a scenario that illustrated the security problem which induced the assessment

process. The user forum case featured an example scenario concerning an increased

need to detect unattended luggage and objects in a mass transportation facility. At

the heart of the proposed solution was a range of advanced computer vision

technology. This could be implemented into the existing CCTV system at the facility -

capable to recognise and track people and objects based on different algorithms

processing the CCTV footage. The technology provider, impersonated by Graeme

Jones of Kingston University, suggested a solution that would invite further

capabilities, with different implications on the technical requirements as well as on a

range of different trust and freedom infringement issues. After creating a new

assessment case with the toolkit and completing the configuration phase together

with the audience, the participants were asked to answer a carefully prepared set of

questions that were presented on their computer.1

3. Summary of results

Feedback on: General

During their trial with the AST, participants were asked to give their opinion on the

overall the structure and presentation of the tool. Because the AST and its

conceptual underpinnings were presented in detail, participants had no problem

understanding the methodology and structure of the tool. Some noted that without

the presentation they may have had more difficulties. This points towards a need to

provide sufficient documentation and help in the toolkit, such as an overview of the

assessment support process implemented in the toolkit2, some of the specific

1 For screenshots please see appendix 2 Configuration phase - assessment phase - reporting phase

4

concepts utilized3, as well as the overall assessment methodology applied. The latter

issue will be addressed in guidelines in the handbook being created in work package

12.

Furthermore experts asked for further definitions of terminology related to scientific

language as well as for the topics and aspects in the respective tasks. In general, the

tool was perceived as a bit unbalanced with regard to the content, as freedom

infringement-related questions were dominant in the set and not all filters were

working yet. As a result of this, it was suggested later on that an alternative way of

navigating through the questions could be provided by means of a graphical

interface which makes transparent the different semantic relations between topics

and tasks.

Feedback on: Functionalities of the AST

The AST shall contain further options and helpful tools to enable users to actually

find answers and use the AST according to its purpose. So far the toolkit which

contains methodologies that have been developed in other work packages of SIAM

(e.g. workshop organization, threat assessments) is located at the top bar of the AST.

The user forum participants saw the need to have the tools and advice present in the

context of each individual question.

One of the key functions of the AST is the possibility of delegating questions to other

AST participants. This function was introduced to make it possible for a user to “work

off” all posed questions, even if the knowledge of the current user is not sufficient

and the methodologies cannot give enough advice to this user to find suitable

answers. The user forum participants very much appreciated this function but

highlighted that it needs more information than just the names of the AST

participants at the delegation tool (currently, only the names of the other actors

were offered for selection as possible recipients of a delegated question). Some

users showed interest in the possibility to implement the internal messaging system

offered by the AST into their own organisation. It was debated whether providing

very detailed information about the other users would encourage AST participants to

3 Tasks – Topic –Aspect – Question

5

delegate questions more frequently if they do not suppose these in their field of

competence - rather than finding useful answers by themselves. This indeed would

reduce the purpose of the AST in increasing reflexivity in the assessment process.

The consortium and the user forum participants discussed that this could be

contained if delegated questions were somehow flagged to the assessment leader

and the reflexivity score reduced.

One more idea put forward was a tool that could arrange a meeting between all AST

participants, which highlighted the necessity of personal contact and coordination in

assessment processes in general.

Feedback on: SMT Typology

The SMT typology was not instantly obvious to some users. Understanding the

typology is critical, especially for the assessment leader, whose task it is to configure

an assessment case. Part of this task is to decompose the technological solution

considered in a case into its technological parts and to assign the appropriate SMT

classes, which then partially determine which assessment questions are provided to

the assessment participants. It was recognized that sufficient documentation and

training needs to be provided to the assessment leaders so that they can perform

this task.

Feedback on: Definitions of Roles

In the configuration phase the assessment leader invites other actors to participate

in the assessment process. To each participant one particular role will be allocated.

This will also determine which questions will be posed to that participant reflecting

his or her professional role and competences. In the current specification, the

consortium decided to allocate only one role to one participant, but practitioners

raised the question that in some companies, especially in SMEs, one person can have

multiple roles in the process. Furthermore, it became evident that a new role is

required to address the problem that some questions are too complex in the judicial

context. Hence, the role of the ‘lawyer’ will be added to target all legal questions

that cannot be answered by participants that have only a general judicial knowledge.

6

Feedback on: Questions

The overall opinion was, that sometimes questions were too complicated in the

formulation, or the level of information required to answer them was perceived too

high. This was often the case when questions about legal compliance were posed.

However, SIAM aims to increase reflexivity such that situations should be avoided

where users can simply ignore questions they do not want to deal with. A solution

for this problem is that the SIAM consortium will establish a set of broader questions

that will be posed to everyone and then subsequent question in respect of deeper

understanding will be assigned to particular roles. In the case of the legal questions

the already mentioned new role ‘Lawyer’ will be introduced. It became also clear

that some questions need further context information to enable users to find the

answers for those questions. However, testing the tool with users exposed that

answering questions with only yes or no will not produce the necessary knowledge.

As a solution, it was considered that available answer options may be enhanced with

an additional text field to justify or explain an answer given. The purpose of this is to

make users shift their answer habits from a normative towards evidence-based

reasoning. The toolkit must provide necessary assistance in the form of

supplementary information such as tools or methodological advices that help users

find answers. A further outcome of the user forum was that the questions must be

very clear in distinguishing between passengers and employees. The frameworks for

these two groups are too different.

Feedback on: Reporting

After the completion of the assessment phase it is the assessment leaders’ task to

summarize all the given answers into a coherent report. The SIAM AST provides a

comprehensive editing tool where the answers given by all participants are

presented, ordered by task, topic, and aspect. The assessment leader can then

formulate summary statements on the various issues which are then printed in the

assessment report. The user forum highlighted the importance that the assessment

leader should be given a guideline on how to summarize all the answers, in order to

make sure all issues are addressed appropriately. Assessment leaders have a

powerful function in the toolkit, because in the end they decide which opinions get

7

emphasized in the report. To balance that role, SIAM introduced an “Observer” AST

function. Users of this role are not directly participating in the assessment, however,

they are able to audit the reporting activities performed by the assessment leader.

There also must be further explanations on the reflexivity scores that indicate how

many divergence roles have come into play in the AST.

4. Conclusion

The second user forum presented a well structured alpha version of the AST. The

suggestions, criticisms and ideas given as feedback at the first user fora were used to

improve the toolkit substantially. Fortunately, the SIAM consortium was able to

acquire some of the participants from the last user fora and supplement them with

renowned experts from leading scientific institutions and other practitioners. This

was the first time where user not related to the project would test the tool and

evaluate its content. This provided valuable insight into their interaction with such a

tool and gave pointers towards issues where SIAM still has to be improved.

All adaptions will take place in the course of the coming months. SIAM will then

present an operational beta version of the AST at the ‘Computers, Privacy & Data

Protection’ (CDPD) international conference in Brussels on January 22nd, 2014.

8

5. Appendix

Screenshots of the Assessment Support Toolkit (AST) presented at the User forum

October 31st, 2013.

Login Screen

Case Dashboard

9

Establishment of a new case

10

11

12

Technology depiction

13

Actor involvement

14

Assessment phase

15