Mutual Empowerment in Human-Agent-Robot Teams

16 December 2010

HART Workshop

Jurriaan van Diggelen

Problem statement

• Achieve more with less people

• Automation can help to:– Make better use of available semi-structured

information sources– Support decision makers in dealing with the

complexity of problems (war amongst the people)

The big number cruncher

• Monolithic approach, BNC replaces existing infrastructure

• AI-complete

Sensor data

Twitter data

UAV images

Problem solution

Towards a human-machine team solution

• Solution must be provided by a human machine team

• Mutual empowerment seeks to improve team performance by:– Compensating weaknesses of humans and

machines– Optimizing strengths of humans and

machines

Types of Mutual Empowerment

human machineHMI

Intelligent Interfaces

human machine

CM

I

CC

I

HMI

User empowerment

DistributedArtificialIntelligence

CollectiveIntelligence

ME handbook

Goal

Methodology

Functional design

PrototypingValidation

•Use cases•Claims•Cognitive requirements•Ontologies•Performance measures•Tests/benchmarks

•System requirements•Functional modules•RDF interface specifications•Prototypes

•Mixed reality validation•Data collection

Tool support

Domain Exploration

•Domain•Human Factors•Technology

Situated Cognitive Engineering

• Methodology supports– Incremental design– Reuse of earlier work (Prototypes, tests,

requirements, use cases) – Collaborative development

Example

Phase 1: domain exploration

• Domain– USAR– UGV, UAV– Operators in field

• Human Factors– Maintaining situation awareness– Cognitive overload– Adaptive teams

• Technology– Collaborative tagging, crowd sourcing– Mixed initiative systems– Adaptive/ adaptable automation

Phase 2: Functional design (1)• Use cases

• Cognitive requirements

• Claims

UC 23• UAV classifies camera image as victim with certainty-level Unsure• Operator of Robot1 is notified of the potential victim and views the

camera images • Operator of Robot1 classifies the image as victim with certainty level Certain• Operator of Robot2 is notified about the victim• …

CR 5.1 Uncertainty managementOperators and agents can publish and change the certainty value of informationUse cases: UC 23

CR 5.1 • + improves situation awareness of operators and agents• - increases cognitive taskload

Phase 2: Functional design (2)

• Ontologies

• Performance measures– E.g. situation awareness measure

• Tests/benchmarks– Test for evaluating performance

something

action event item

victimrobot

Phase 3: Prototyping

• Develop system requirements that implement the cognitive requirements.

• Bundle system requirements in functional modules.

• Reuse existing base platform

Trex

Trex• Filter: which

data do you want to see? selection of semantic tags in Sparql

• Projection: How do you want to see the data?graphical object with attachment-points for semantic tags

Functional modules supported by Trex

• User configurable information filters• User configurable information visualization• Realtime semi-structured data exploration• Collective relevance assessment• Uncertainty management• Human-in-the-loop AI

P Q R S THuman Machine Crowd Machine

Human-in-the-loop AI

DEMO

Future work

• Develop functional modules for:– Joint conflict resolution– Adaptive Interruptiveness– Network awareness– Policy awareness– Capability awareness– Activity awareness

Conclusion

• Mutual Empowerment library provides a flexible way to– Increase application possibilities of AI– Employ potential of collective intelligence– Reuse and structure our knowledge of

human-machine collaboration tools

Technology InvestigationDomain Analysis Human Factors

Metrics

Cognitive Requirements

ClaimsUse cases

OntologiesTests

Exploration

Functional design

Prototyping

Core functions

Functional modules

System Requirements

PrototypeRDF interfaces

TestingSimulation Test participants Empirical results