Conversational

Sensemaking

Alun Preece, Will Webberley

(Cardiff)

Dave Braines (IBM UK)

The International Technology

Alliance

2006–2016: Fundamental US/UK research into Network and

Information Science to support coalition operations.

Our ongoing research is funded by US Army Research Labs

and the UK Ministry of Defence.

see http://usukita.org

Introduction

The story so far

• Human-centric sensing (2012)Srivastava, M., Abdelzaher, T., & Szymanski, B. (2012). Human-centric sensing. Philosophical

Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 370(1958), 176-197.

• CE-SAM: a conversational interface for ISR mission support (2013)Pizzocaro, D., Parizas, C., Preece, A., Braines, D., Mott, D., & Bakdash, J. Z. (2013, May). CE-SAM: a conversational interface for ISR mission support. In SPIE Defense, Security, and

Sensing (pp. 87580I-87580I). International Society for Optics and Photonics.

• Human-machine conversations to support multi-agency missions (2014)Preece, A., Braines, D., Pizzocaro, D., & Parizas, C. (2014). Human-machine conversations to support multi-agency missions. ACM SIGMOBILE Mobile Computing and Communications

Review, 18(1), 75-84.

• Conversational sensing (2014)Preece, A., Gwilliams, C., Parizas, C., Pizzocaro, D., Bakdash, J. Z., & Braines, D. (2014,

May). Conversational sensing. In SPIE Sensing Technology+ Applications (pp. 91220I-91220I). International Society for Optics and Photonics.

• Conversational sensemaking (2015)

Pirolli & Card

“The sensemaking process for intelligence

analysis”

Foraging loop

• Gather and assemble data,

present as evidence

• Less focus on structure

and formality

Sensemaking loop

• Schematize evidence,

connect to hypotheses

• Inform decision making

• Support sharing and

presentation of insights

Reimagining Data-to-Decision

Data sources are increasingly “smart” and communicative

Decision-makers can operate much nearer to the tactical edge

Humans can be sensors too; and effectors when appropriate

Analytic services Decision maker Data sources

The traditional data-to-decision pipeline can be re-thought

as peer-to-peer interactions between human and machine

agents with different specialisms and focus areas

Back to Pirolli & Card

We envisage the sensemaking process underpinned by

a conversational interaction between teams of human

and machine agents.

• Supports forward and

backward flows

• Provides some structure

from the start

• A less segmented view

of the world?

• Enables co-construction

of information artifacts

• Structure can increase as

the conversation evolves

Human-Machine

Conversational Model

Background: Format for

conversationAn appropriate form for human-machine interaction is a

challenge:

humans prefer natural language (NL) or images

these forms are difficult for machines to process, leading

to ambiguity and miscommunication

Compromise: controlled natural language (CNL)

there is a person named p1

that is known as ‘John Smith’

and is a person of interest.

low complexity | no ambiguity

ITA Controlled English (CE)

Our conversational model

• Draws on research in agent communication languages

and philosophical linguistics (speech acts)

• We envisage valuable conversations between:

– Human and machine

with mediation between Natural Language (NL) and CE to

allow unambiguous but human-friendly exchanges

– Machine and human

asking the human for more information or

informing them of relevant details as appropriate.

Often “gist” (computed NL) form is useful here

– Machine and machine*

Exchanging information between software

agents and/or pre-existing systems.

Use of CNL enables easier human oversight

ask/tell

confirm

why

gist/expand

* Also human and human, but that is not covered here

Bag-of-words NLP

• The purpose of the conversational

interaction is to allow humans to use

natural language (NL)

• NL is converted to CE through

simple “bag of words” NL processing

– Consult the knowledge base for

matches and synonyms

– Covering the model (concepts,

relations, rules) and the “facts”

• Confirmation of interpretation can

(optionally) be sent to the user

– Confirmation is in CE; the machine

format but human readable

– Not always appropriate to share

• Model can be expanded through the

conversation too

Examples of conversation

• In our ongoing research we have applied our

conversational interactions to the following

scenarios:

– “SPOT” reporting

– Crowd-sourced information

gathering

– Asset tasking

– Hard/soft information fusion

• The potential benefits could include:

– Improved agility

– Reduced training

– Improved effectiveness for

human/machine hybrid teams

• Harnessing the power of each type of agent

Conversational Foraging

Introducing MOIRA

• “Moira” – Mobile Intelligence Reporting

Application

• A machine agent able to engage in

conversation

• Access to CE knowledge base

– Can read all available knowledge,

explore and answer questions

– Can help the human user contribute new

knowledge

• Model, fact, rule

• Contextual operation

– Aware of the users role, location, status

– Able to alert “interesting” information

Three initial experiments20 untrained student participants viewed a

series of scenes and described them to

Moira via confirm interactions

• 137 NL scene descriptions in 15min

• Median CE elements per NL input = 2

39 untrained student participants

crowdsourced answers to 54 questions re

synthetic and natural situations in multiple

locations

• 718 NL inputs yielding 479 CE inputs in

30 min

• 69% of users had > 1 point

18 members of the public crowdsourced

answers to 30 “television trivia” questions

at a BBC festival event

• 101 NL answers yielding 62 CE

confirms

Histogram of score

frequencies

Enriching the shoebox

• The shoebox is central to the foraging loop

• A “messy” store of information drawn from

external data

• Our “semantic” shoebox:

– Contains data from multiple sources

– NL and CE

– Some low-level schema exist

– Able to extend the schema at run-time

– Human (or machine) users can add new data or new sources

– Inferences can be made

– Rationale and provenance can be available

• This semantic shoebox can be iteratively refined

– From low -> high value CE

– Serving the sensemaking loop too

– Can store hypotheses, presentation models and much more

A sensemaking blackboard

• This “semantic shoebox” is actually a sensemaking

blackboard

– An open “sandpit” blackboard; not task/solution specific

• The agents:

– Human users

• Define/extend the model

• Capture local knowledge & insight

• Direct agent activities

– Machine agents

• Execute logical inference rules (general)

• Existing software algorithms (specialised)

• Control through triggers, alerts, commands etc

• The single language is ITA CE, with “rationale” for

explanation

• CEStore & CENode implementations

NATO protest example• Prior to the event we modeled protests

and events

• Instances can be added by any

agent conceptualise an ~ event ~ E that

has the time ST as ~ start time ~ andhas the time ET as ~ end time ~ and

~ involves ~ the agent A and~ is located at ~ the place P.

conceptualise a ~ protest ~ P that

is an event.

there is a protest named ‘Central Square protest’ that

has the time 4-9-2014-12:00 as start time andinvolves the group ‘Blue Group’ and

is located at the place ‘Central Square’.

• During the event we unearthed the important difference

between expected and unexpected protests

– Real-time model update was made

– Rule was written to detect unexpected protests

– Alerting of unexpected protests

• Sometimes they can be

detected from text

analysis of tweets

Conversational Sensemaking

Blurring the boundaries

• In Pirolli & Card the distinction between foraging and

sense-making is clear

• Distinct interactions between the loops are possible

• Human and machine tasks are acknowledged but separate

• Through conversation and our

“blackboard” approach we:

– Support rich multi-agent integration

– Enable flows between different loops and

phases

– Grow the shoebox upwards

– Drive (some) schema downwards

• Agile models & human-friendly

formats to encourage more active

participants

Adding context

During our field exercise we noted that:

• Key influencers can be identified

• Data relating to events can be found

• A range of possible values may be

presented (e.g. for crowd size)

• Conscious and subconscious biases

may be present

Approaches to identify (and potentially

quantify) biases exist

• We modeled “stance” for key influencers

• Pro-NATO and Anti-NATO

• Knowledge of this “stance” is important

contextual knowledge for human

observers

and machine agents

This is a good example of closing the

loop from hypothesis to data-collection

Moving to richer models

• We can “grow the shoebox” as we progress

to higher levels

• Rather than “increased schematization” we

introduce richer models, or refine models

through conversation

• Hypotheses can be modeled, subjectivity can be captured (or

computed)

– Including propagation through inference or other computation

• Rationale (asking “why?”) can link higher level models to lower

level information

• Related work:

– “Collaborative human-machine analysis using a controlled natural

language” – Mott et al

– Argumentation, trust and subjective logic

Presenting through storytelling

• Apply narrative framings to the body of

knowledge

• Also expressable in CE

– A generalised abstraction of storytelling that can

be applied to any domain

– Organising the domain into an episodic

sequence

– Applying additional multi-modal information

• Using connected hypotheses, evidence and

data to tell a story

• Asking “why?” to uncover rationale for

information

Wrapping up

Summary

• Envisage Pirolli & Card feedback loops as a series of

human-machine conversations

• Helping to harness each agents strengths?

– Humans: Interpreting & hypothesising

– Machines: large scale data, pattern collection

• Rationale to promote transparency and trust

• Enabling debate and argument– Reveal conflicts (and agreements)

– Explore (and maybe reconcile) differences

• Currently focus is on text communications

• Future experiments:

– Mix of human and machine-based sensing

– Grow links with argumentation research

Conversational Sensemaking

Originally presented at:

SPIE DSS 2015 – Next Generation Analyst III

(Human Machine Interaction)

Any questions?

Research was sponsored by US Army Research Laboratory and the UK Ministry of Defence

and was accomplished under Agreement Number W911NF-06-3-0001. The views and

conclusions contained in this document are those of the authors and should not be

interpreted as representing the official policies, either expressed or implied, of the US Army

Research Laboratory, the U.S. Government, the UK Ministry of Defense, or the UK

Government. The US and UK Governments are authorized to reproduce and distribute

reprints for Government purposes notwithstanding any copyright notation hereon.

Many of the examples in this paper were informed by collaborative work between the authors

and members of Cardiff Universities Police Science Institute, http://www.upsi.org.uk. We

especially thank Martin Innes, Colin Roberts, and Sarah Tucker for their valuable insights on

policing and community reaction.