DARPA and Data: A Portfolio Overview - Reason · 2019-02-04 · 1 DARPA and Data: A Portfolio Overview William C. Regli Special Assistant to the Director Defense Advanced Research

1

DARPA and Data:A Portfolio Overview

William C. RegliSpecial Assistant to the Director

Defense Advanced Research Projects Agency

Brian PierceDirector

Information Innovation OfficeDefense Advanced Research Projects Agency

Fall 2017

Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

2

• Investments over the past decade span multiple DARPA Offices and PMs• Information Innovation (I2O): Software Systems, AI, Data Analytics• Defense Sciences (DSO): Domain-driven problems (chemistry, social science,

materials science, engineering design)• Microsystems Technology (MTO): New hardware to support these processes

(neuromorphic processor, graph processor, learning systems)• Products include DARPA Program testbeds, data and software

• The DARPA Open Catalog• Testbeds include those in big data, cyber-defense, engineering design, synthetic

bio, machine reading, among others• Multiple layers and qualities of data are important

• Important for reproducibility; important as fuel for future DARPA programs• Beyond public data to include “raw” data, process/workflow data

• Data does not need to be organized to be useful or valuable• Software tools are getting better exponentially, ”raw” data can be processed• Changing the economics (Forensic Data Curation)

• Its about optimizing allocation of attention in human-machine teams

DARPA Dreams of Data


3

Working toward Wisdom

Abstraction(also“value”)

Sapience (or processing/cognitive ‘load’)

Information

Knowledge

Wisdom

Understanding

data

Wisdom: sound judgment - governanceUnderstanding: explanation – whyKnowledge: correlation and composition – howInformation: facts - what, who, where, whendata: representing signals or stimuli

Example of the Data-Wisdom process for model evolution and application:Information Knowledge Understandingdata

Data (signals)

from multiple sources

WisdomIndigenous Forces

Insurgent Forces

Indigenous Government

InfrastructureInfrastructure

International Diplomacy

Coalition Forces

Local Civilians

Insurgent SympathizersCoalition Population

Media Coverage

Descriptive model(symbols, syntax)

Correlative/predictive model(rules, processes,

semantics)

Explanatory model(abstractions, theories, etc)

MILITARY MILITARY --TACTICALTACTICAL

MILITARY MILITARY ––STR / OP STR / OP COALITIONCOALITION

PUBLICPUBLIC

COALITIONCOALITIONGOVERNMENTGOVERNMENT

MEDIAMEDIA

LOCALLOCALCIVILIANSCIVILIANS

IRAQI IRAQI GOVERNMENTGOVERNMENT

IRAQI POLICE/IRAQI POLICE/MILITARYMILITARY

INSURGENTSINSURGENTS

Public COINUndersta nding

and Expectati ons

Empower-ment of

ProgressiveLocal Le aders

Security &PerceivedStability

Inform ationOps

Effectiveness

Quality & Availabilityof COIN Messa ge

Qual/Availof Local I nfo

CollectedTarg eted

Offensive OpsEffectiveness

GovtSupport

for Conflict

Manpow erFundin gEquip

Adequacy

Govt COINAppropri ate

Actions

Govt COINUnder-

standing &Expect-ations

Military toGovt Comms

Quality

Dev Fundi ngAdequacy

& TimelinessManpow er

perPopulationAdequacy

Logistics OpsEffectiveness

WORKING HYPOTHESIS

IRAQ COUNTER-INSURGENCY DYNAMICS

© PA Knowledge Limite d 200 6. All rights reserv ed. – 10/1 8/200 6

Insurge ntAbility toProvokeExcessiveResponse

SituationalStress

MilitaryAppr Useof Forc e

Instit’lCOIN

Learni ng& AdjustBureaucr acy &

Organizati onalDelays

Insurge ntTarg etedThre ats &Attacks on

Iraqis

CivilianCasualties & Damages

Fear ofInsurge nt

Repercussio n

LocalKnowledge

CoalitionFear ofMistake

Insurge ntCasualties

and Capt ures

CoalitionCasualties

Infrastr uctur e &Econ Dev OpsEffectiveness

RelativeAttractive-ness of

Insurge ntPath

Media Susceptibilityto Insur gent

Message

Media Emp hasis on Negativ e &Sensational

MediaSensation-alism Bias

Interacti onOpport unity/Consistency Military to

CivilianTrust

PositiveCivilian

Interacti onsFractio n

Emphasison CivilianPerceptions

& Relationships

CivilianWillingnessto Inter act

Appropri ateInter-

person alBehavior

CivilianInsurge ntSymp-athizerFractio n

Ability to IDInsurge nts &

PreventAttacks

MilitaryCivilianContacts

Depth & Qualityof MediaCoverag e

PerceivedMilitary

Use of Fo rceSatisfaction

withDaily Life

PerceivedPoliticalFairness

Perceptionof CoalitionIntentions &Commitme nt

PerceivedReligious &Ideological

Compatibility

ElapsedTime ofConflict

Hope forSelf

Governe dFutur e

ExpectedBenefits

to Daily Life

Visible Benefitsto Daily Life

Infrastr uctur eEconomy

Employment

Insurge ntRecruiting& Retention

Insurge ntCapability

Insurge ntManpow er &Leade rship

Insurge ntMaterialSupport

Hard Cor eInsurge nts

in Area

COINApprSkills

Instit’l COINUnder-

standing &Capability

Avg COINCapabilityper Soldie r

Recent COINRelated

Experience

Inform alCOIN

Learni ng& Adjust

Qual/Availof COINTrainin g

COIN ApprTools

COINSpecialistAdequacy

COIN EffectiveResourceAllocation

& Ops Focus

Proactive COINAppr Men tality

& Focus

MilitaryExposure toInsurge nt

Attack

Qual/Avail ofInsurge ntMessage

CivilianParticipationin Central

Govt Events

Iraqi Cent ral Govern ment

Effectiveness and Cre dibility

CoalitionPerceivedConflictCosts Public

Supportfor Conflict

Insurge nt Abilityto Oper ate

in Area

CivilianProx-imity

Ability toSurviveInsurge ntAttack &Retaliate

Insurge ntAttacks

on Military

Insurge ntLearni ng

LocalPresence

OpsEffectiveness

Guardin gOps

Effective-ness

Ability to Vet Relig/Trib/Polit/MilitLeade rs

Effective Useof Existing

Govt/Police/Milit Capab

Unit Duratio nin Area

& RIP/TOAQuality

Consistencyof MilitaryActivity/

Personality

Tribal &ReligiousTension

Local GovtDevelopme ntCredibility &Effectiveness

Contracto rAdequacy

ISF IPManpow er

Effectiveness& Credibility

Local Govt/Police/Military

Dev OpsEffectiveness

Quality & Volume ofActionable

Intel

IntelAnalysis

OpsEffective-

nessRelative

Effectivenessof Insur gentMessage on

Civilians

Judgement, value,insight, instinct, principles, etc



PUBLICPUBLIC


MEDIAMEDIA






and Expectati ons

Empower-ment of



Inform ationOps

Effectiveness



CollectedTarg eted


GovtSupport

for Conflict


Adequacy


Actions

Govt COINUnder-



Quality





WORKING HYPOTHESIS




SituationalStress


Instit’lCOIN




Iraqis


Fear ofInsurge nt

Repercussio n

LocalKnowledge



and Capt ures

CoalitionCasualties



Insurge ntPath


Message




CivilianTrust

PositiveCivilian



& Relationships


Appropri ateInter-

person alBehavior



PreventAttacks



PerceivedMilitary


withDaily Life




Compatibility


Hope forSelf

Governe dFutur e

ExpectedBenefits

to Daily Life



Employment






in Area

COINApprSkills




Recent COINRelated

Experience

Inform alCOIN

Learni ng& Adjust


COIN ApprTools



& Ops Focus


& Focus


Attack



Govt Events




Supportfor Conflict


in Area

CivilianProx-imity


Insurge ntAttacks

on Military

Insurge ntLearni ng

LocalPresence

OpsEffectiveness

Guardin gOps

Effective-ness





& RIP/TOAQuality


Personality



Contracto rAdequacy

ISF IPManpow er





Intel

IntelAnalysis

OpsEffective-

nessRelative


Civilians


Hybrid operations

Indigenous Forces

Insurgent Forces

Indigenous Government

InfrastructureInfrastructure

International Diplomacy

Coalition Forces

Local Civilians

Insurgent SympathizersCoalition Population

Media Coverage

Descriptive model Correlative/predictive model Explanatory model



PUBLICPUBLIC


MEDIAMEDIA






and Expectati ons

Empower-ment of



Inform ationOps

Effectiveness



CollectedTarg eted


GovtSupport

for Conflict


Adequacy


Actions

Govt COINUnder-



Quality





WORKING HYPOTHESIS




SituationalStress


Instit’lCOIN




Iraqis


Fear ofInsurge nt

Repercussio n

LocalKnowledge



and Capt ures

CoalitionCasualties



Insurge ntPath


Message




CivilianTrust

PositiveCivilian



& Relationships


Appropri ateInter-

person alBehavior



PreventAttacks



PerceivedMilitary


withDaily Life




Compatibility


Hope forSelf

Governe dFutur e

ExpectedBenefits

to Daily Life



Employment






in Area

COINApprSkills




Recent COINRelated

Experience

Inform alCOIN

Learni ng& Adjust


COIN ApprTools



& Ops Focus


& Focus


Attack



Govt Events




Supportfor Conflict


in Area

CivilianProx-imity


Insurge ntAttacks

on Military

Insurge ntLearni ng

LocalPresence

OpsEffectiveness

Guardin gOps

Effective-ness





& RIP/TOAQuality


Personality



Contracto rAdequacy

ISF IPManpow er





Intel

IntelAnalysis

OpsEffective-

nessRelative


Civilians

Select and execute interventions



PUBLICPUBLIC


MEDIAMEDIA






and Expectati ons

Empower-ment of



Inform ationOps

Effectiveness



CollectedTarg eted


GovtSupport

for Conflict


Adequacy


Actions

Govt COINUnder-



Quality





WORKING HYPOTHESIS




SituationalStress


Instit’lCOIN




Iraqis


Fear ofInsurge nt

Repercussio n

LocalKnowledge



and Capt ures

CoalitionCasualties



Insurge ntPath


Message




CivilianTrust

PositiveCivilian



& Relationships


Appropri ateInter-

person alBehavior



PreventAttacks



PerceivedMilitary


withDaily Life




Compatibility


Hope forSelf

Governe dFutur e

ExpectedBenefits

to Daily Life



Employment






in Area

COINApprSkills




Recent COINRelated

Experience

Inform alCOIN

Learni ng& Adjust


COIN ApprTools



& Ops Focus


& Focus


Attack



Govt Events




Supportfor Conflict


in Area

CivilianProx-imity


Insurge ntAttacks

on Military

Insurge ntLearni ng

LocalPresence

OpsEffectiveness

Guardin gOps

Effective-ness





& RIP/TOAQuality


Personality



Contracto rAdequacy

ISF IPManpow er





Intel

IntelAnalysis

OpsEffective-

nessRelative


Civilians

Tactical operations

Detect and identify objects/entities

Recognize relationships and activities of objects/entities

Find endo– and exogenous causal factors driving objects/entities

Assess viability of Blue approach

Examples of Data-to-Wisdom

4

TaskInformationDetermine facts

(what, who, where and when)Correlate and compose(networks and systems)

KnowledgeExplain

(causal mechanisms)

UnderstandingApply sound judgement

(governance)

Wisdom

Information operations

Message translation Message meaning Evolve message strategyMessage impact

Security operations

Image object identification Scene knowledge(relationships between objects)

Students at a university walking between buildings

Scene understanding

Students walking because of a change in classes

Decide and manage responses

Students walking because of a change in classes is not anomalous behavior



DARPA Investments in the Data Ecosystem

LORELEI (I2O)MoDyl – Dynamics from data sets (DSO)RATS (I2O)Visual Media Reasoning (I2O)Global Autonomous Language Exploitation*

Brandeis – Data privacy (I2O)EQUIPS – Uncertainty analysis tools (DSO)Memex – Web data/info search (I2O)SAFER (I2O)Simplex - Unified math frameworks (DSO) XDATA (I2O)PROCEED*Signal processing programs*

CASCADE – Integ data/tools for robust SoS (DSO)CRAFT – Workflow/CAD tools (MTO)Data Driven Discovery of Models (I2O)Deep Extraction from Text (I2O)Insight (I2O)Make-It – Synthetic chemistry (DSO)MENTOR2 – CAD data/tools (DSO)Modelling Adversarial Activity (I2O)Network Defense (I2O)Next Gen Soc Sci – Data/tools (DSO)QCR (I2O)Sigma – System for CBRNE detection (DSO)TRADES - Eng design env (DSO)Personalized Assistant that Learns*

Cortical Processor (MTO)EdgeCT (I2O)GRAPHS – Graph analysis tools (DSO)HIVE – Graph problem HW/MW (MTO)Media Forensics (I2O)

MUSE (I2O)SafeWare (I2O)

Deep Learning*

AIDA (I2O)Big Mechanism (I2O)Biochronicity – Tools gene regulatory nets (DSO)Causal Exploration (I2O)MATRIX– Multi-scale matls data/models (DSO)Mine Better, Fund Faster – Surprise ID (DSO)SocialSim (I2O)Synergistic Discovery and Design (I2O)

BRASS (I2O)Explainable AI (I2O)FunLOL – Framework for AI/ML (DSO)MSEE – Tools for machine understanding (DSO)PPAML (I2O)

*completed program

InformationDetermine facts


Explain(causal mechanisms)

Knowledge Understanding

Serv

ices

& M

iddl

ewar

eAl

gorit

hms

& An

alyt

ics

Serv

ices

& M

iddl

ewar

eAl

gorit

hms

& An

alyt

ics


DARPA Investments in the Data Ecosystem

LORELEI (I2O)MoDyl – Dynamics from data sets (DSO)RATS (I2O)Visual Media Reasoning (I2O)Global Autonomous Language Exploitation*

Brandeis – Data privacy (I2O)EQUIPS – Uncertainty analysis tools (DSO)Memex – Web data/info search (I2O)SAFER (I2O)Simplex - Unified math frameworks (DSO) XDATA (I2O)PROCEED*Signal processing programs*

CASCADE – Integ data/tools for robust SoS (DSO)CRAFT – Workflow/CAD tools (MTO)Data Driven Discovery of Models (I2O)Deep Extraction from Text (I2O)Insight (I2O)Make-It – Synthetic chemistry (DSO)MENTOR2 – CAD data/tools (DSO)Modelling Adversarial Activity (I2O)Network Defense (I2O)Next Gen Soc Sci – Data/tools (DSO)QCR (I2O)Sigma – System for CBRNE detection (DSO)TRADES - Eng design env (DSO)Personalized Assistant that Learns*

Cortical Processor (MTO)EdgeCT (I2O)GRAPHS – Graph analysis tools (DSO)HIVE – Graph problem HW/MW (MTO)Media Forensics (I2O)

MUSE (I2O)SafeWare (I2O)

Deep Learning*

AIDA (I2O)Big Mechanism (I2O)Biochronicity – Tools gene regulatory nets (DSO)Causal Exploration (I2O)MATRIX– Multi-scale matls data/models (DSO)Mine Better, Fund Faster – Surprise ID (DSO)SocialSim (I2O)Synergistic Discovery and Design (I2O)

BRASS (I2O)Explainable AI (I2O)FunLOL – Framework for AI/ML (DSO)MSEE – Tools for machine understanding (DSO)PPAML (I2O)

*completed program

InformationDetermine facts


Explain(causal mechanisms)

Knowledge Understanding

Serv

ices

& M

iddl

ewar

eAl

gorit

hms

& An

alyt

ics

Serv

ices

& M

iddl

ewar

eAl

gorit

hms

& An

alyt

ics

DARPA XDATA

Batch (size)

Stream (rate)

Radar

Speech

Video

Text

Spreadsheets

Metadata

PCAP

IP Networks

Bio

Etc…

Disk / RAM / GPU

Data In

gest API

Tikka,OO

DT‐W

ings W

orkflow (M

DA/JPL), Zep

hyr (Sotera), Op

tiWrangler(Stanford O), Othe

r ETL

HDFS / HadoopMap / ReduceGiraphOozieHamaRHIPEHiveGoFS

USCAccumulo

BlinkDB

Storm

Spark Streaming

Mesos

TachyonSparkSharkBagel

Java

HiveQL

PythonAnacondaBlazeNumba

Clojure

SNAPDeliteOptiCVXOptiGraphOptiML

g++

CUDABigdata DB

MPI

ElementalBLASOpenMPI

Single Machine

HPC

TBD

GPU

gpuGraph

RMatlabStataRiposteSNAP

New School Defaults and design recommendations

Boeing / U Pitt Genie

Continuum Stencil DSL

HTML5

d3.js

Vega

Aperture

OWF / Neon

Disk / RAM / GPU

Sources

UCB

StanfordGA TechIBMContinuum

JHUStanfordPhronesisCMUPurdueStanford O

PurdueUSCSoteraMITLLData Tactics

Continuum

Bokeh

Graph / Matrix

IBM Skylark(RNLA)Sotera Louvain USC BSP OptimizationGA Tech NNMFJHU VNOMStanford O SNAP

Time SeriesSotera ARIMAGA Tech Time SeriesSSCI/MIT/UL BayesDB

Machine LearningSSCI/MIT/UL BayesDBMITLL/BBN NLPBoeing/Upitt SMILEMITLL TopicIDDT/UMD/MSR

ClassificationUCB BlinkDB

Distance Measures

TBD e.g, Cosine Sim, Page rank

GA Tech / Stanford B / Continuum Convex Optimization

SoteraCorrelation

Approximation

Royal Caliper

UC DavisSYSTAP

Platform Abstractions Analytics

Data Analytics API

Lincoln LabsA

PI. H

iveQ

LAPI, O

ODT‐Wings W

orkflow (M

DA/JPL)

Ad‐Hoc R

etrieval A

PI

Mon

go, Impala, Shark (UCB

), OO

DT‐W

ings W

orkflow (M

DA/JPL)

Stanford O

RAM / Cache / GPU

Visualization

Stepi+1

UCB CherryPy

Tangelo

Next Century

Oculus

Kitware

Stanford / Kitware

Continuum Logging API

Attention Tracking

/ Eye Tracking

/ Us

er Interfa

ce Action Tracking

via keystroke logging and mou

se click

ing(Drape

r)

Interactive Querying

Search / Que

ry / Subselect/ Sum

marize

/ Lin

k‐Brush / P

ivot / Zoom

/ Expo

rt / Integrate / Intersect

“Find me everything on this number”

“Find me a pattern of behavior on this entity”

“Where will this person be tomorrow”

“What groups does this group interact with”

Transforms

Tracklets

Speech to text

Content extraction

Entity ExtractionTopic Modelling

Pcapunpacking

Geocoding

Other basic transforms

SQL‐Like

Graph‐Like

Geo/Time‐Like

Oculus / Kitware

Oculus / Kitware

Oculus / Kitware

Interaction Queries

Virtual EnvironmentsImmersive

3d

Physical Devices

(Multi‐touch) ICT

Workflow Management / System ArchitectureMDA/JPL, UCB, Sotera, Draper, Data‐Tactics

Quality of ServiceLatency from clickRAID and redundancy; #GPUs, RAM, SSD, CPUAdjustable tradeoffs: Volume, Time, Accuracy / Uncertainty

Resource Management/Planning Dependencies; Scheduler Performance and load estimatesUser Interface with workflow planning

ICTInteractive Coding

(Wakari, Julia)

MITLL / Continuum

1

7

X-Data: Common Data Cyber-Infrastructure


8

Examples: Human-Machine Symbiosis

37

DARPA’s Transformative Design (TRADES)

Program ObjectiveTransform design by exploring new math/algorithms to enable new and innovative designs which account for next generation materials and processes

Expected Outcomes• Leverage computation and data at

unprecedented scales• Deal with significantly more complex

and coupled design problems• Lower the time and expertise required

to create complex engineered products



Summary

Unbound Computation & DataExploit computation and larger scales; leverage machine learning and AI;Produce and harvest data in more disruptive ways! ß requires culture change

Advancing the software toolboxAn ecosystem of interoperable building blocks/tools for machine-augmented problem solving; make them easier to use ß a limiting issue

Problem-Process-People Co-designSimultaneous redesign of problem and human-machine dynamic:Thinking faster, better and functioning at higher levels ß culture change

Information

Knowledge

Wisdom

Understanding

Abstraction

Sapiencedata

Today Future

Information

Knowledge

Wisdom

Understanding

Abstraction

Sapiencedata

9Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

www.darpa.mil

10Distribution Statement “A” (Approved for Public Release, Distribution Unlimited)

11

A DARPA Goal: Transform cognitive problems using machines-as-partners

Credit: Detroit Institute of Arts


12

Big Mechanism

Goal: Machines that help humans to model and analyze very complicated systems by reading fragmented literatures and assembling and reasoning with models.

Courtesy: Rzhetsky Lab


13

Big Mechanism Architecture

journalarticles text shallow and with models

model manager

reading

entities, relations, events, processes models assembly

suggested modelrevisions

revised models

Gene OntologyReactome

Pathway Commonsetc ...

reasoningpredict, explain, test, curate, etc.

Goal: Machines that help humans to model and analyze very complicated systems by reading fragmented literatures and assembling and reasoning with models.

Outline of today’s talk: Reading; Some Results; Varieties of Big Mechanism Systems; Why it Works; Will it Work Elsewhere?


• Altered Senses• New sensing systems; new human-machine interaction as well• Neurotech being coupled with higher-order cognitive processes

• Unbound Computation & Data• Exploit computation and larger scales;

Produce data and harvest our own in more disruptive ways!• Advancing the software toolbox

• An ecosystem of building blocks/tools for machine-augmented problem solving; make them easier to use ß a limiting issue

• Problem-Process-People Co-design• Simultaneous redesign of problem and human-machine dynamic

Cross-cutting Opportunities for DARPA


Distribution Statement A – Approved for Public Release, Distribution Unlimited 15

System Concept

Shared Services

Knowledge Organization

Human Model Interaction User

Shared Repository Common Standards and Interfaces Scalable Processing

Causal Model

Assembly

Integrated Causal Model

Entities, Causal

Relationships, Domain-tailored

Ontology

User Input and Feedback to

Model Assembly

Model Assembly/ Population/ Configuration/ Adaptation

ICM Exploration

Intuitive Interaction

Knowledge Needs/Gaps,

Request/ Response

Operational, Government-provided, and Open SourceDocuments

and Databases