A Testbed for Studies of Team A Testbed for Studies of Team Cognition in the Cyber Security Cognition in the Cyber Security

DomainDomain

A Testbed for Studies of Team A Testbed for Studies of Team Cognition in the Cyber Security Cognition in the Cyber Security

DomainDomain

Nancy J. CookePrashanth RajivanShankaranarayanan Venkatanarayanan

Arizona State University5 May 2010

Background• Education: Cognitive Psychology/Human Factors

George Mason University, B.A. New Mexico State University, M.A., Ph.D.

• Positions Rice University New Mexico State University Arizona State University & Cognitive Engineering Research Institute

• Applied Experience: U.S Air Force, Navy, Army, NASA, NTSB, VA

• Section Editor, Human Factors• USAF Scientific Advisory Board• National Research Council Committee on Human

Systems Integration

Relevant Research

Team Cognition Military, Cyber, and Medical Applications

Communication Analysis

Metrics for Coordination and CollaborationSponsors

• Air Force Office of Scientific Research• Air Force Research Laboratory• Office of Naval Research• Army Research Office • Leonard Wood Institute• Veteran’s Administration – MWM VERC

Cooke’s Background

Overview

• MURI and ASU Team• Team Cognition and Team Situation

Awareness• Other Team Testbeds• CyberCog – New Testbed

MURI: Computer-aided Human Centric Cyber Situation Awareness

DoD Multidisciplinary University Research Initiative (MURI) program project, funded through Army Research Office

Two fundamental limitations of Cyber Situation Awareness (C-SA)• Gap: human cognition < -- > C-SA tools

– Situation data exceeds “cognitive throughput” of human analysts• “Blind spots” in views of cyber situation for existing C-SA tools (including

auditing, vulnerability scanners, attack graph tools, intrusion detection systems, damage assessment tools, and forensics tools)

Cyber-SA Vision• Build data < -- > human decision links through innovations

– knowledge fusion– cognitive automation– artificial intelligence– visual analytics

• Awareness-driven cyber defense vs. malware behavior dependent defense • Automatic blind spot identification and monitoring techniques 　

MURI Partners• Professor Peng Liu, Penn State University, Overall PI • Professor Nancy Cooke, Arizona State University • Professor Coty González, Carnegie Mellon University • Professor Dave Hall, Penn State University • Professor Sushil Jajodia, George Mason University • Professor Mike McNeese, Penn State University • Professor Peng Ning, NC State University • Professor VS Subrahmanian, Univ. of Maryland • Professor John Yen, Penn State University • Professor Michael Young, NC State University

ASU MURI TeamNancy J. CookeProfessor, Cognitive Science & EngineeringCollege of Technology and Innovation

Prashanth RajivanGraduate Student Master’s in Computing StudiesCollege of Technology Innovation

Shankaranarayanan VenkatanarayananGraduate Student Master’s in Computing StudiesCollege of Technology and Innovation

Teams and Cognitive TasksTeam is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide,

perceive, design, solve problems, and act as an integrated system.

Cognitive activity at the team level= Team Cognition

Improved team cognition Improved team/system effectiveness

Heterogeneous = differing backgrounds, differing perspectives on situation

(surgery, basketball)

• Unmanned Aerial Vehicles • USS Vincennes shoots down

Iranian airbus (1988)• Challenger/Columbia accidents

tied to poor organizational decision making (1986/2003)

• Response to 9/11 reveals communication breakdowns (2001)

• Katrina response lacked coordination (2005)

• Sago Mine disaster report cites poor command-and-control (2006)

• VA Tech communications substandard (2007)

• Friendly fire incidents• Various health care mishaps

attributed to poor teamwork

Some Instances of Failures of Team

Cognition

Miracle on the Hudson

And some successes…

Response to Fargo flooding

Interactive Team Cognition in a Nutshell

Team interactions often in the form of explicit communications are the foundation of team cognition

ASSUMPTIONS

1) Team cognition is an activity; not a property or product

2) Team cognition is inextricably tied to context

3) Team cognition is best measured and studied when the team is the unit of analysis

US 2004 Olympic Basketball Team

"We still have a couple of days, but I don't know where we are," replied USA head coach Larry Brown to a question Wednesday on where his team was in its preparations. "We have good moments and bad, but I've got a pretty good understanding of who needs to play. Now the job is to get an understanding of how we have to play."

A team of experts does NOT make an expert team

Collaborative skill is not additive

US 1980 Olympic Ice Hockey Team

Herb Brooks and 20 young “no-names” won the 1980 Olympic Gold Medal in Ice Hockey

An expert team made up of no-names…

In our UAV STE three operators must coordinate over headsets in order to

maneuver their UAV to take pictures of ground targets

Our UAV Testbed

UAV-STE: Uninhabited Air Vehicle (ground control station) Synthetic Task Environment for research on team cognition (DURIP 1997; USAF funded)

Air Vehicle Operator controls UAV airspeed, heading, and altitude and monitors air vehicle systems

Payload Operator controls camera settings, takes photos, and monitors camera systems

DEMPC navigator, mission planner, plans route from target to target under constraints

Interdependence requires interaction, communication, & coordination

Three team members with inter- dependent tasks

Our MacroCog (Macro-Cognition Testbed)

MacroCog Testbed

Navy-funded lab for strategic planning and decision- making in the context of noncombatant evacuation operations

MacroCog Roles in Current Experiment

Information Warfare Specialist

Personnel Specialist: Military

Equipment Specialist:

Land/Sea Vehicles

Personnel Specialist:

Humanitarian

Equipment Specialist: Air

Vehicles

Experimenter 1

Experimenter 2

Example of Empirical Results on Team Cognition

As teams acquire experience, performance improves, interactions improve, but not individual or collective knowledge

0

100

200

300

400

500

600

1 2 3 4 5 6 7 8 9 10

Mission

Team

Per

form

ance

Tm 1

Tm 2

Tm 3

Tm 4

Tm 5

Tm 6

Tm 7

Tm 8

Tm 9

Tm 10

Tm 11

• Individuals are trained to criterion prior to M1• Asymptotic team performance after 4 40-min missions (robust finding)• Knowledge changes tend to occur in early learning (M1) and stabilize• Process improves and communication becomes more standard over time

40-min missionsSpring Break

Team Situation Awareness

A team’s coordinated perception and action in response to a change in the environment

How can we exercise team SA in a testbed?

How can we measure it?

How can we intervene to improve it?

Contrary to view that all team

members need to “be on the same page”

What is Meant by Coordinated Perception and Action?

Measure of Team Situation Awareness• Change is introduced (communication breakdown, enemy in area,

storm) that will impact mission

• 2-3 team members are presented cues regarding change

• Team members need to perceive cues in a coordinated way (i.e., connect the dots) to identify the change

• Team members coordinate to take action relevant to the change (e.g., change altitude, communicate indirectly)

• Measure in terms of outcome and process – who on team was involved?

CyberCog Simulator

Web based Simulator application for measuring individual interaction and team collaboration (e.g., team situation awareness) in a Cyber security analysis situation

CyberCogSimulator – System Overview

CyberCogSimulator – Components

• Cyber Security Analyst (User)– Assigned a specific role such as Denial of Service

(Dos) specialist, Malware specialist and Phishing specialist

– Understands the scenario given, use events and attack symptoms, collaborates with other participants to identify a potential attack or a combination of attacks

– The team reaches a common consensus on the type of attack and its corresponding events

CyberCogSimulator – Components

• Master controller and Evaluator– Queries attack scenarios, events and symptoms

from the database– Distributes the events and symptoms to the

participants– Logs the interaction between participants at real

time– Evaluates and scores the participants findings

with the expected results

CyberCogSimulator – Components• Database server

– MySQL database server stores :-• Attack Scenarios• Events corresponding to attack scenarios

including some false positives & noise events• Attack Symptoms for each specialization (E.g.,

Dos, Malware , Phishing) identified• The expected results, interaction (between

participants ) logs and attack conclusion arrived at by each team for each session

User and Team Views

Legends

FunctionsFunctions

Data

CyberCog Simulator- Interaction

CyberCogSimulator- Architecture

Microsoft IIS

Database

Intra/Internet Malware Specialist

Phishing Specialist

Dos Specialist

Client Tier

Controller & View Tier

POCO’s

ADO.net

Web Services

Model Tier

Conclusion

• There are current gaps and limitations in Cyber Situation Awareness

• Cyber situation awareness by teams involves the coordinated perception and action in the face of a change in the cyber situation

• CyberCog will allow the MURI team and others to better understand team-based cyber SA and to test algorithms and tools developed for improving it

Team Cognition Research Program

Testbeds: 1) UAS C22) Navy Strategic Planning

Empirical Studies in Testbed

UAS Field Data

Theory Development

ACT-R Model of Synthetic Teammate

Dynamical Systems Modeling

Measures

3540 3560 3580 3600 3620 3640 3660 3680 3700

2940

2950

2960

2970

2980

2990

Time (s)

Cum

ulat

ive

Spe

akin

g (s

)