ELICIT Experimental Laboratory for Investigating

Collaboration, Information-sharing, and Trust

2What is ELICIT?

• ELICIT = Experimental Laboratory for Investigating Collaboration, Information-sharing, and Trust

• U.S. DoD (OASD/NII) Command and Control Research Program (CCRP) sponsored the design and development of the ELICIT platform for experimentation focused on information, cognitive, and social domain phenomena

• Purpose of ELICIT-related Experimentation and Analysis is to investigate the cognitive and social impacts of C2 approach and organizational structure (e.g. information sharing, trust, shared awareness, and task performance)

• Initial applications focus on a comparison of traditional hierarchical and edge C2 approaches

3ELICIT Scenario

• The goal of each set of participants is to build situational awareness and identify the who, what, when, and where of a pending attack– Participants can share factoids directly with each other or post factoids

to websites

– Participants build awareness by gathering and analyzing factoids and interacting with one another

• Participants receive factoids about a future attack– Factoids fall into four task categories: who, what, when, and where

– Factoids are periodically distributed to the participants

– No one is given sufficient information to solve their assigned problem without receiving information from others

• The receiving, sharing, and posting of factoids and the nature of the interactions between and among participants can be constrained

4Experiment Details

• Sixty-eight (68) total factoids: – 17 when; 17 where; 17 who; 17 what

– Factoids may be key, supportive, or nonessential

• Factoids are distributed in three waves– 34 factoids at start

– 17 factoids at 5 minutes

– 17 factoids at 10 minutes

• C2 approach for this series of experiments were designated prior to the start of the run as– Hierarchy or Edge

• 17 participants make up an organization• Output:

– Transaction log

– Scratch paper collected

– Survey conducted at the conclusion of the run

Hierarchy

Edge

5Website Access

• The commander (gray) can access/post to all four websites

• All other participants can only access/post their group’s website

• Each participant can access/post to all four websites

Hierarchy Edge

Access to all websites

Where

When

Who

What

Team Member

Website

6Illustrative Factoids and Identifies

Who Factoid: The Lion is known to work only with the Azur, Brown, or Violet groups (classification – key fact)

What Factoid: Bloggers are discussing the role of financial institutions in oppressing the Coral, Violet and Chartreuse groups (classification – supportive fact)

When Factoid: The Brown group needs time to regroup (classification – nonessential fact)

Where Factoid: The Azur, Brown, Coral, and Violet groups have the capacity to operate in Tau, Epsilon, Chi, Psi and Omega-lands (classification – supportive fact)

Sample Identify: "who=Red Group; what=TV station; where=B-land; when=Nov 8, 10:00 pm“

Sample Correct Solution: The Green group plans to attack a TV station in A-land on 25 Nov at 11AM

7Variables of Interest

Info Sharing &Collaborative

Behaviors

SharedInformation

Quality of Information

Shared Awareness

Quality of Awareness

SharedUnderstanding

Quality of Understanding

TaskPerformance

TaskDifficulty

Measures of Merit

NetworkCharacteristics& Performance

Individual& Team

Characteristics

Culture

Allocation of Decision Rights

Quality of Information

Sources

Patterns of Interaction

Distribution of Information

C2 Maturity Level

Partially Controllable

Controllable

Legend

8Dependent Variables

• MOE = Quality of Awareness and Shared Awareness– Correctness (Authorized Correct IDs)– Timeliness (Person-Minutes with Correct IDs)– Accuracy rate (Correct IDs/Total IDs)

• Efficiency, Given Effectiveness– Productivity (Correct IDs/Total Actions; Correct IDs/Person-

Minutes Available)– Speed (Time of Earliest Correct ID)

• Agility– Effectiveness over problem difficulty

9C2 Approach Independent Variables

• Hierarchy v. Edge– We expect Hierarchy to map to De-conflicted and Edge to map

to a more mature level– Each run will be mapped to a point in the C2 Approach Space

based on observed behaviors

• Rules of Interaction– Website access– Sharing permissions

• Initial Distribution of Factoids – Invariant in existing runs

10C2 Approach Intervening Variables

• Patterns of Interaction – Characteristic path length– Clustering coefficient– Connectedness

• Distribution of Information– The average number of unique facts to which each participant

has access as a function of time

11Measures of C2 Effectiveness (MOCE)

• Quality of Information Position– Percentage of relevant facts for the assigned task that a

participant can access as a function of time– Percentage of key facts for the assigned task that a participant

can access as a function of time

• Extent of Shared Information– The average number of participants that have access to each

fact as a function of time– The average number of participants that have access to each

key fact as a function of time

12Intervening Behavioral Variables

• Activity over time (sharing, website posts, website pulls, ID attempts)– Sharing

• Peer-to-peer sharing• Posting

– Information Seeking• Pulling

– Identification Attempts

13Other Independent Variables

• IDs allowed• ELICIT experience of the player• Factoid set (problem difficulty)• Translated factoids v. original• Native Language (English v. Other)• Communications media

– Postcards– Chat

• Time available• Degree of Education (Graduate, Undergraduate)• Seniority (Rank)• Subcultures (Military, Civilian, Special Forces, Civil

Servants)

14ELICIT Data Set

• Includes data from 37 ELICIT experimentation trials• Venues

– Boston University (2 runs)

– Naval Postgraduate School (16 runs)

– Portugal (6 runs)

– US Military Academy (3 runs)

– Singapore (10 runs)

• Organization Types – 18 Edge

– 19 Hierarchy

• Past excursions include– Chat capability

– Varying the subjects (Intel, Military, PhD)

Future use planned for Germany, UK, SAS-065, NDU, AWC

15Extracting Data from ELICIT

• ELICIT generates output in a number of forms– Participant survey responses and “scratch paper”

– Transaction logs recording all information exchange and identify actions

• While rich in data, transaction logs must be processed in order to be formulated for analysis– Transactions of each type must be extracted and tabulated

– Identify attempts must be graded for correctness

• Scripts (Python, C++) and macros (Visual Basic) were employed to facilitate data structuring for analysis– Software is available for use by others

• Transactions can be parsed and organized into time intervals to enable observation of how behaviors change over time, and of the timing of identifies– Whole-trial statistics also taken to characterize trials, but difficult to

compare due to differences in trial duration