Methods and Metrics for Analysis of Sensemaking
Dr Karen Carr & Mr Barry McGuinness
BAE SYSTEMS
Advanced Technology Centre
Our objectives
• For this meeting:
– Contribute what makes sense to us, in our given context and with our goals
• In our work:
– Develop the ability to supply C3I ‘capabilities’ (in partnership)– Systems Engineering of socio-technical systems– Driven by need to deliver usable and demonstrable results – Science as well as engineering and domain expertise
NB We want to ensure that human issues drive the developments - but we don’t want to forget that we have to
inform technology (as well as organization, process)
What we mean by sensemaking
Why we want to use this concept to try and answer the questions we need to answer
• Our question:
“How can we develop technology, design and manage systems which support/ enhance the human roles in defence operations?”
– Significant human role is ability to adapt, respond to unexpected, creativity, play mind games, etc. Need to preserve & enhance that - not interfere.
– Support is needed for dealing with the unexpected, the unknown, as well as recognisable situations
– Include broad System of Systems issues, developers, rapid change
Sensemaking (& Situational Awareness) is a working concept to enable us to start manipulating, analysing, and measuring context, goals and human performance
Why we want methods and metrics for studying sensemaking
• Need to attribute effects - predict - in order to provide support.
• Move from concepts to metrics to analysis to (testable) models.
• Reduce subjective bias (influence of our own sensemaking, interpretation)
• No existing clear metrics we can use - no absolutes
1. Understand how human performs, and what conditions facilitate ‘good’ performance (what hinders)
2. Identify the properties of organisation, process, technology, training, etc which are important for success
3. Develop design and management methods and tools to enable implementation
• NB not necessarily numbers - could be properties
Range of methods
• observation (non intrusive)
• subjective investigation (e.g. ethnographic, knowledge elicitation)
• storytelling/anecdotes (knowledge building)
• metaphor (pattern matching)
• scientific method (empirical hypothesis testing)
• mathematical analysis (baseline)
Methods and Metrics
• Concepts
• Metrics
• Some analyses
• Implications for sensemaking
Concepts
• Orientation
– complex, uncertain situations– SA determines capacity to decide and act– sensemaking determines SA– cognitive processes are intrinsically goal-directed– people form nested hierarchy of processes & outcomes
• Objectives
– Understand SA and sensemaking– Feed into design & development of information systems and
human systems– Applied research -- theory into practice
What is a Situation?
• A situation is a pattern in state space, especially one which appears to deviate from a “normal” intended or expected pattern.
• Example:- aircraft fuel x time into flight
Aircraftfuellevel
Time intoFlight
Unexpected rate-- we have a situation!
Normal takeoff
Normal cruise
Unrecognized Patterns
• An unrecognized pattern demands attention.
Attention!
Perceivedpattern
???
Knownpatterns?
Unknown pattern?
perception comprehension
Definitions
Knowledge:
= capacity for “action”
Situational Awareness:
= dynamic “situated” knowledge, i.e. capacity to act effectively here & now in a given specific situation
Sensemaking:
= process of creating effective SA in situations of uncertainty
“Knowing what’s going onso you can figure out what to do.”
• doing• saying• thinking
Situational awareness
Dynamic mental representation of the current and future state of one’s domain of action
– includes awareness of • environment• entities • events • processes • actions • others’ perceptions & intentions
– insofar as these are relevant to • performing an action, or • choosing a course of action
Through a continuous process of situation assessment
Situational awareness
SA is based on ...
• prior KNOWLEDGE– SCHEMAS: generalized patterns representing typical situations– based on experience, training, culture
• recent INFORMATION– direct perception of the environment– perception of instruments and displays – received communications
SA
INFORMATION
KNOWLEDGE
PHYSICAL DOMAIN
instruments communications
Central role of SA
SA
Decision-making
Actionperformance
PHYSICAL DOMAIN
Informationacquisition
Sense- making
• SA both informs and is informed by • sense-making• decision-making
COGNITIVE DOMAIN
Inside SA: Cutting up the cake
Concrete(situation-specific)
Abstract(generalized patterns)
Observed Implied
Information
specific propositionse.g., “rate of fuel
loss is high”
Intentions
selected actions afforded by situation
e.g., “Contact ATCand inform”
Models
situational schemase.g., “Fuel leak?”“Faulty sensor?”
Projections
mental simulationse.g., “Risk of not
reaching destination”
Processes involved in SA
PERCEPTION Acquisition of information about
the given situation
COMPREHENSION Diagnostic interpretation of
the given situation
PROJECTION Prognostic simulation of future situations
and their possible outcomes
RESOLUTION Selection of actions to direct the given
situation towards the desired outcome
… All serving to support dynamically effective action
information
models
intentions
projections
Sensemaking and SA
Information
Models
Intentions
Projections
PERCEPTION
COMPREHENSION PROJECTION
RESOLUTION
Decisionmaking
Sensemaking
Sensing Acting
Sense-making: when comprehension is uncertainDecision-making: when resolution is uncertain
Metacognition
• Defined as:
– “Thinking about thinking” or “knowledge about knowledge”– i.e. “Awareness of your own SA”
• noticing uncertainties, gaps, conflicts in your mental reps
• identifying information needs
• employing strategies for sensemaking & decision-making
SA
“It’s like looking over your own
shoulder.”
?
Gives a subjectivesense of SA
SA and metacognition
• Four possible states:
AppropriateConfidence
(ideal state)
InappropriateConfidence(danger state)
InappropriateSensemaking
AppropriateSensemaking
True SA False SA
Confident in SA
Not confident in SA
Actual awareness:
Need for sensemaking
Subjective attitude
Team SA and shared SA
• Not the same thing
• Team SA = sum of current knowledge held across a team, irrespective of who has it
• Shared SA = those parts of the team SA that are common between team members
Shared SA
Team SA
Personal SA
What to share, with whom?
• The nature of SA in groups is dictated by goals
• Goals are hierarchic
• Top-level goals are shared by all members
– therefore need shared SA with respect to that objective
• Lower-level goals are specific to individuals
– therefore need personal SA with respect to own task
• Sharing one’s SA is necessary only to the extent that the knowledge has bearing on the goals of others
Team SA and shared SA
• Shared SA elements can be differentially allocated
perception
perception
perception
comprehensionprojection
resolution
perceptionresolution
perceptionresolution
perceptionresolution
projection
comprehension
Distributed SA in the C2 HQ
Ops
Signal Commander
Intel
Intentions(RESOLUTION)
Information(PERCEPTION)
Models(COMPREHENSION)
Metacognition?
Projections(PROJECTION)
So...
• Explicit sensemaking processes are needed when comprehension cannot easily occur
• Sensemaking requires metacognitive awareness of own knowledge -- uncertainties, gaps
• Metacognitive assessments can be wrong and lead to inappropriate subjective attitude -- and inappropriate behaviour
Measuring SA
• COGNITIVE approach
– queries about the situation – Reveals mental reps
• Multiple choice (SAGAT)• True/False (QUASA)• Sit Reps
• SUBJECTIVE approach
– self-ratings of SA – Reveals metacognitive state
• Unidimensional (SARS)• Multidimensional (SART)• Multidimensional and intelligible! (CARS)
• OBJECTIVE approach– behavioural & physiological correlates – Reveals changes in metacognitive state
• EEG, fMRI• Eye pointing
As a rule, take both cognitive & subjective measures together.
CARS
• Crew Awareness Rating Scale
• a subjective tool to elicit operator’s subjective sense of SA
• multi-dimensional
• generic, adaptable, easy to use
Dimensions
Knowledge Processing
Perception
Comprehension
Projection
Resolution
Eight CARS questions
1. the most recent information2. what is really going on here3. what could happen4. what actions should be taken
1. monitor the flow of information2. understand the big picture3. predict how it is likely to evolve4. decide what actions to take
Would you say you have a good sense of …
Would you say it is easyfor you to …
knowledge
processing
Six possible responses
YES
NO
Certain Uncertain
Definitely
Definitely not Think not
Think so
For sure?
Don’t know
Don’t need it
Do I ?
CARS results
CONTENT• Perception ||| ||| | |• Comprehension | |||| || |• Projection || |||| | |• Resolution | ||| ||| |
PROCESSING• Perception |||||| ||• Comprehension ||| ||| ||• Projection || || ||| |• Resolution || |||| ||
Def Prob Prob not Def not DK NA
CARS results
0
40
80
20
60
% o
f ra
tings
100
Run 1 Run 2 Run 3 Run 4
1. Definitely
2. Probably
3. Probably NOT
4. Definitely NOT
1. Definitely
2. Probably
3. Probably NOT
4. Definitely NOT
Comprehension knowledge over time
QUASA
• Quantitative Assessment of Situational Awareness
• a probe tool to elicit operator’s actual SA
• mathematical : based on SDT
• still under development, but promising
QUASA
• Signal Detection TheoryYES!
perception discrimination
Square?
• Targets vs non-targets• Hits, False Alarms, Good Misses, False Rejections• Also applies to internal (mental) representations
QUASA
• “Is this item true?”
– Confidence in perceived truth value of items varies
Confidence in truth value of items
Num
ber
of it
ems
Max SENSITIVITY= ideal SA
Weak Strong
TRUEitems
FALSEitems
QUASA
Confidence in truth of items
Num
ber
of it
ems
No sensitivity,poor SA
Weak Strong
Literally can’t tell the difference between true & false items…
They have similar-strength levels of confidence
QUASA
Confidence in truth value of items
Num
ber
of it
ems
Max NEGATIVE sensitivity= worst case SA
Weak Strong
FALSEitems
TRUEitems
Deception
QUASA
Confidence in truth value of items
Num
ber
of it
ems
SA’
IB’’Weak Strong
Some positive sensitivityLow positive bias
(acceptance threshold)
Bad acceptancesBad rejections
Good rejectionsGood acceptances
QUASA
Example probe:“ The tanks adjacent to bridge are enemy ”
Response: YES (accept as true) or NO (reject as false)
Evaluate:Sensitivity (discrimination of true/false situations) = SA’Bias (probability of item acceptance/rejection) = IB’’
QUASA
Maximumpositive
sensitivity:ideal SA
Maximumnegative
sensitivity:the wrong situation!
Maximumpositive
bias:too rash
Maximumnegative
bias:too cautious
TYPICAL
+100
0
-100
QUASA
Perception: information
Comprehension: model of situation
Projection:Future
developments
Resolution:CoA intention
+100
0
-100
Dynamic SA - D
-100
-80
-60
-40
-20
0
20
40
60
80
100
4 5 6 7 8 9 10 11 12 13 14 15 16 17
Last turn of block
SA' score
Info bias (IB'')
QUASA
• Mathematical assessment of SA
• Needs the truth!
• SA, bias, components, temporal
• ? Team & shared SA
Behavioural correlate of SA
Tracking eye-point-of-gaze (EPOG)
Do EPOG patternscorrelate with SA?
CRT CRT CRTCRT
ECAM(engines)ND & EFISPFD
FMS
throttles
touch panel(optional for electroniccharts, datalink)
touch panel(systems control display)
COCKPIT LAYOUTforward view
displays and controls
ND & EFIS PFD
DisplayControl
Unit
Navcontrolssidestick
priorityPR ALT
SPDALT
HDG
VSA/P
on/off
mode control panel(Flight Control Unit)
spoilers & flaps indicators
LDG GEAR controls
flapsspoilers
sidestick
Commscontrol
unit
ALT autopilot altitude controls & indicatorA/P on/off autopilot master on/off toggleCRT Cathode ray tubeECAM Electronic Centralised Aircraft MonitoringEFIS Electronic Flight Instrumentation SystemFMS Flight Management SystemHDG autopilot heading controls & indicatorLG GEAR landing gearND Nav DisplayPFD Primary Flight DisplayPR ALT pressure (altimeter) setting & indicatorSPD speedVS autopilot vertical speed controls & indicator
ECAM(systems)
EPOG research
‘Entropy’ = known loss of SA
SA and flightdeck automation
Collision avoidance system
+01
Heathrow control this is Speedbird five five, descending now to flight level one four.
Speedbird five five, Heathrow control, roger, descending to flight level one four.
Heathrown control this is Delta four zero four, flight level two zero, request descent clearance.
Delta four zero four, Heathrow control, what is your present altitude?...
Radio “party line”
SA and flightdeck automation
Traffic Situation Reporting
Detectableaircraft
Non-Detectableaircraft
50%
25%
0%
With automation
Conventional
Reportedaircraft
SA and C2 digitization - ISTAR
Own force positionsEnemy positions
BGHQ crewstationCommon Operational Picture
SA and C2 digitization - ISTAR
Battlespace digitization demonstratorSynthetic
environment
SA and C2 digitization - ISTAR
•2-hr ISTAR recce operation
•Performed with voice AND digital C2 systems
Measures taken of mental workload & situational awareness
SA and C2 digitization - ISTAR
rating
voicedigital
aspects of SA (knowledge of enemy)
PERCEPTION
DEF
PROB NOT
DEF NOT
PROB
PROJECTION RESOLUTIONCOMPREHENSION
Some implications
Both actual SA and subjective sense of SA affect decision-making & performance
Technology can affect SA for better or worse
Analyses with metrics provide specific insights
Other work
• DS1 trials
• BattleLab trials
• Cognitive modelling
– COGNET in C2 environment– Ideal Decision Maker
• ? Can be used to predict dips in SA and sensemaking needs
Building Industry-MoD partnership
Implications for sensemaking
• Thinking about thinking
• Concepts : sensemaking as processes supporting SA
• Role of metacognition : group context
• Metrics of SA : can be used to evaluate sensemaking solutions
• Data can feed development of predictive models
• Knowing what’s going on so we can figure out what to do!