Comparing two techniques for intrusion visualization

Vikash Katta1,3, Peter Karpati1, Andreas L. Opdahl2, Christian Raspotnig2,3 & Guttorm Sindre1

1) Norwegian University of Science and Technology, Trondheim2) University of Bergen, Norway

3) Institute for Energy Technology, Halden, Norway

Andreas.Opdahl@uib.no

The ReqSec Project

Method and tool support for security requirements engineering:

involve non-experts lightweight integrated, add-on industrially evaluated

Funded by the Norwegian Research Council (NFR), 2008-2012

Many techniques proposed, e.g., anti-behaviours...

Perspective

System security models:

black-box models of monolothic systems

single systems security analysis and specification

Security architecture models:

high-level organisational views

enterprise architecture for security

Need for intermediate solutions:

security modelling for SOA

white-box models of service collaborations

bordering organisation and technology

Misuse Case Maps (MUCM)

Inspired by Use Case Maps (R.J.A. Buhr, D. Aymot...)

Misuse Case Maps (MUCM)

Use case maps:

components, scenario paths, responsibilities

Misuse case maps:

vulnerabilities, exploit paths, vulnerable responsibility

Preliminary evaluations:

good for architectural overviews

need better visualisation of attack step sequences

Misuse Sequence Diagrams (MUSD)

Misuse Sequence Diagrams (MUSD)

Sequence diagrams:

actor, object/component, action, event/message

Misuse sequence diagrams:

attacker, vulnerability, exploit action and event/message

Initial evaluation:

can MUSD complement MUCM?

how do the two techniques compare wrt.• understanding• performance• perception

Comparison

Controlled experiment with 42 subjects

Latin squares organisation, random assignment

Treatment (independent variables): technique: MUCM, MUSD task: bank intrusion (BAN), penetration test (PEN)

Measures (dependent variables): understanding (UND) performance (VULN, MITIG, VUMI) perception (PER)

Control (control variables): background (KNOW, STUDY, JOB)

Hypotheses

H11: MUCM better on architectural questions

H21: MUSD better for temporal sequence questions

H31: Either technique better on the neutral questions

H41: Either technique better overall

H51: Different numbers of vulnerabilities identified

H61: Different numbers of mitigations identified

H71: Different total numbers of vulnerabilities and

mitigations identifiedH8

1: Usefulness perceived differently

H91: Ease of use perceived differently

H101: Intentions to use perceived differently

H111: MUCM and MUSD perceived differently

Procedure

4 groups of 10-11 2nd year computer science students

10 steps:• Filling in the pre-experiment questionnaire (2 min)• Reading a short introduction to the experiment (1 min)• First technique on first task:

introduction to the technique (9 min) read about task, looking at diagrams (12 min) 20 true/false questions about the case (8 min) finding vulnerabilities and mitigations (11 min) post-experiment questionnaire (4 min)

• Easy physical exercise (2 min)• Repeat for second technique and task (44 min)

Results

Backgrounds: No sig. differences between groups:

Kruskal-Wallis H test– 2-4 semesters of ICT studies– 2.07 months of job experience (three outliers)

Sig. knowledge differences across groups:– Wilcoxon signed-rank tests– KNOW_MOD > KNOW_SEC, p = .000– KNOW_SD > KNOW_UCM, p = .003– KNOW_MUSD ≈ KNOW_MUCM

Understanding

Wilcoxon signed-rank tests

H1 & H2 accepted, H3 & H4 rejected

Medium effect size (Cohen)

No impact of technique or task order

Performance

Two blank outliers removed (from 11-student groups)

H5, H6 & H7 rejected

No impact of technique order

More identifications for bank task

Perception

H8, H9, H10 & H11 accepted

Medium to large effect sizes (Cohen)

Only one insig. statement (“would be useless”)

More positive perception of first technique used

Conclusion

The techniques are complementary

They facilitate understanding better for their “intended use”:– MUCM best for architectural issues– MUSD best for temporal sequences

They are equal in performance– the bank task was more productive

MUSDs were perceived more positively– the first technique was perceived more

positively

Further work: simpler MUCMs, qualitative analysis, more techniques, industrial subjects, notation and method integration, industrial case studies and action research...

Central concepts

RFC 2828:

vulnerability: a weakness in a system ... that can be exploited to violate its security policy

threat: a potential for violation of security ... that could cause harm

countermeasure: something that reduces a threat or attack by eliminating... preventing... minimizing the harm... or by reporting it to enable corrective action