Matthew Rizzo, MD, FAANFrances and Edgar Reynolds Professor & Chair

Department of Neurological Sciences

Clinical Program Leader, Neurosciences

Director, Great Plains IDeA CTR

Supported by

U.S. NIA, NINDS, NHLBI,, NIDA, CDC, Nissan,

Toyota CSRC & State of Nebraska

Mind and Brain Health

MS&V for Predictive Leadership and Decision-

Making

September 25, 2018

Overview

• Neuroergonomics

• Simulation

• Linking Physiology and Behavior

• Brain-in-the-Wild

• Drinking from a Firehose

• Conclusions

IDeA-CTRs

Center for Research Capacity Building, National Institute of General Medical Sciences,

National Institute of Health. (2016) IDeA-CTR Principal Investigator Directory 2016.

Retrieved from https://www.nigms.nih.gov/Research/CRCB/IDeA/documents/2016IDeA-CTR_Directory.pdf

Neuroergonomics

Key Issues and Approaches•Cognitive Readiness

•Decision Making and Stress

•Lessons from Brain Damage

•Bridging the gap between lab

and real world

•Simulation

•Experience Sampling

•Triangulating on Behavior

Effects of fatigue, drugs, brain injury, disease, aging

Information processing and the brain

• Analyze individual operators’visual perceptual and attentional, memory, executive and motor functions, and emotions.

• Assess performance with simulations, in the field and past performance records

• Evaluate which cognitive profiles impairments underlie specific operator errors and adverse outcomes.

• Develop training and task related countermeasures.

Lessons from brain injury

Inouye, 1905

*FDA here!

Localizing functional errors

•Emotion execution sites include hypothalamus, basal forebrain, and brain stem (tegmental nuclei).

Emotion comprises visceral, and endocrine and somatomotor responses (not shown).

Visceral sensations reach anterior insular cortex via the brain stem.

Feelings result from re-representing visceral changes in relation to the object or event that incited them. The anterior cingulate cortex is a site of this secondary mapping.

•Information on emotionally

salient objects is represented

in brain sensory systems.

•This information can be

derived from the environment or recalled.

•It is made available to trigger

sites for emotion: amygdala &

orbitofrontal cortex.

•There are feedback connections: Do you run

away because you are

frightened or vice versa?

The Frontal Lobes

• Comprise about 30% of the entire cortical surface

• Can be grossly divided into dorsolateral, medial, orbital and polar regions.

• These regions are organized into networks that include, cortical, subcortical, and limbic.

• They are damaged in a wide variety of conditions

• Are critically important to decision-making in real-life, including in military decisions.

Traumatic Brain Injury (TBI)

• Arises from projectile wounds, blunt trauma and acceleration-deceleration forces (e.g,

vehicle crashes, falls, blast injuries.

• TBI is associated with

– diffuse axonal shearing of white matter

– cerebral contusions, lacerations and hemorrhages

» petechial

» subdural

» intraparenchymal)

» often involving frontal and anterior temporal lobes

Neuropsychological Tests

• Tests of vision and cognition

• Standardized

• Easy to use

• Widely available

Benton Visual Retention Test (VRT)

Trail-making test, Part B

•The TMT is a widely used and reliable measure of executive functions. •On Part B, the subject must learn to simultaneously track two types of information, which requires cognitive flexibility, planning, and executive switching of attention between tasks. •The TMT is well standardized, easy to administer and score, and time-efficient.

RED

GREEN

Stroop: Name the color of the ink

Participants see names of colors written in colored ink

Must name the color of the ink instead of reading the name of the color.

Provides a measure of behavioral control and resistance to interference.

Awareness

• The Awareness Interview assesses discrepancy between a patient's self-reported cognitive problems

and measurement of those abilities with standardized neuropsychological measures (Anderson and Tranel, 1989).

• Lack of awareness of self-impairment is a common problem neurological disorders, fatigue and stress.

• These individuals are more likely to place themselves

in harm’s way despite being less able to perform

Value of Simulation

• Assessment

• Training

Simulators -Advantages

• Provide the best means to replicate exactly

the experimental task conditions under which

operator decisions are made

• Simulations are safe, without the injury risks

of the real world.

• Can be used to quantify performance profiles

in context in at risk situations /cohorts.

– eg, hostage situations (ARL), driving simulation in

risky drivers (teen, neurological disease, drugs)

The Holodeck: Introduced in the first episode of

Star Trek: The Next Generation (1987-1994)

•VR tools range

from surrealistic

gaming

platforms to

caves to fully

immersive

HMDs.

•These computer

based synthetic

environments

are used for

training, treating

and augmenting

human

performance

1

10

RIGHT - LATERAL LEFT - LATERAL

LEFT - MEDIAL RIGHT - MEDIAL654321123456

Subject

Oncoming Vehicles

HonkingVehicle

5 S

4 S

3 S

3

4

5

6

7

8

9

vmPFC BDC NC

*

**

DECISION-MAKING

Siz

e o

f gap for 50%

accepta

nce

CONTROL

SOCIALPRESSURE

Susceptibility to social pressure following VMPF damage Chen, Rusch, Dawson, Rizzo, & Anderson, Social

Cognitive & Affective Neuroscience, 2015

• vmPFC damage more likely to select smaller & potentially unsafe gaps under social pressure

• Comparison group gap selection did not change under pressure. • vmPFC damage showed prolonged elevated SCR under pressure. • Comparisonsshowed similar initial elevated SCR which then declined prior to left-turn decisions

SENSEI

Simulator for Ergonomics Neuroscience, Safety

Engineering & Innovation

Simulation Infrastructure

• Advanced data collection

capabilities

• Full integration with a network of

• Scenario authoring tools

• Software to allow real-time

monitoring of human behavior

• Channels for real-time graphic

rendering

• Servers to support data

analysis and storage

SENSEI

Linking Physiology and Behavior

Potential Physiologic Measures

•Body temperature

•Blood Pressure

•Heart rate

•Respiratory rate

•Galvanic skin response

•Electroencephalographic activity

•Electromyography activity

•Gastric activity (Electrogastrogram, EGG)

•Eyelid closure (an index of alertness and arousal)

•Eye movements and other motor control measures

•Oximetry (blood oxygenation- earlobe or fingertip)

•Transcranial doppler sonography

•fNIRS

•Blood/serum measures of metabolites or drugs

•Cerebral metabolic activity (PET, fMRI)

•Combination with human brain lesion method

Drowsy Operators

Head Nod

Microsleep before head nod

5 sec Microsleep

Simulation differs from the real-world

Behavior in the wild

Measuring behavior and

physiology in the field, in

naturalistic settings

Sensors and Power

Experience Sampling Method (ESM)

• Attempts to provide valid tools to describe variations

in self-reports of mental processes & behavior

• Provides data on frequency, patterns and intensity of

– daily activity, social interaction, and movement;

– psychological (emotional, cognitive) and conative

dimensions (mental processes or experiences geared

toward action, e.g., impulse, desire, volition, and striving).

– thoughts, including quality and intensity of thought

disturbance.

• Can be applied to study a range of issues in normal

and clinical populations, including in military settings.

Experience Sampling Method (ESM)

• ESM measures correlate with

• physiological measures

• baseline psychological tests

• behavioral indices

• Csikszentmihalyi & Larson (1987). Validity and

reliability of the Experience-Sampling Method J

Nerv Ment Dis, 175, 526-536.

37

• Subjects respond to queries when alerted

– researcher not present

– alerted several times per day

– business as usual until alerted

– duration often 1-3 weeks

• Need tools for

– alerting participant (random, timed, event

based/user-driven)

– delivering queries

– capturing participant responses

Experience Sampling Method (ESM)

38

What a participant might carry

Tools for capturing responses include: paper booklets,

PDAs, Phones, audio recorders, cameras.

39

Video validation of report

photos - of interest to the participant or that document

what the participant was experiencing or doing

Brain in the WildDiagnosis, Treatment, and Monitoring Outside

Hospitals and Clinics • Ubiquitous embedded sensors (in

devices, vehicles, buildings, clothing and

people) monitor behavioral and

physiological data continuously, over

extended time frames

• Data from this IoT together with ecological

momentary assessment and self-report

allow unprecedented observation of the

brain in the wild.

• We can discern heretofore unknown

behavior phenotypes linked to acute and

chronic dysfunction in neural systems.

• Results show novel relationships

between brain, behavior, mobility, and

health.

• Connect to EMR, advance CTR (NIH, etc.) Internet of Things (IoT)

• Driver with our Black Box System brakes rapidly to avoid

hitting a child who ran in front of her car (left panel).

• GPS data (middle panel) locates the car as the child runs

into its path.

• Sensor data capture the rapid vehicle deceleration (right

panel, speed).

• Longitudinal g force negative spike as vehicle brakes (right

panel, acceleration).

Predictions of Driver Safety in Advancing Age (NIA R01 AG17177)

LINKING GLUCOSE AND DRIVING

SAFETY

Unsafe Driving EventSeverely hypoglycemic

Drinking from a Fire Hose

• Naturalistic data sets can be

huge

• Putting it all together-

synchronizing data streams

• “Drinking from a firehose”

– Data Analysis Strategies:

Data mining/machine

learning, pattern

recognition, SAX, Principal

Component Analyses, etc.

– Data storage and

confidentiality (HIPPA, etc.)

• Combine methods to study a phenomen. (Denzin, 1978)

• Multiple viewpoints increase accuracy.

• Metaphor from geometry, navigation and military strategy using multiple referents to pinpoint targets.

• Behavior research can improve accuracy of findings by considering multiple sources of data on human performance and safety.

Triangulation

Simulation

Instrumented Vehicle

Black Box/Naturalistic

“truth"

Four triangulation

types

1. Data: data are

collected across

different times, people,

situations, etc.

2. Investigator: multiple

researchers gather and

interpret data

3. Theoretical: multiple

theoretical frameworks

are used to interpret

data

4. Methodological

multiple methods are

used to collect data.

Triangulation in mind and brain research

Telehealth: diagnosis, treatment and monitoring outside hospitals and clinics

/Analytics

• Symbolists: Focus on inverse deduction. Instead of the classical model of

starting with a premise and looking for conclusions, inverse deduction starts

with a set of premises and conclusions and works backward to fill gaps.

• Connectionists: Focus on re-engineering the brain. Key example is Deep

Learning”. Approach is based on connecting artificial neurons in a neural

network. Connectionist techniques are efficient in areas such as image

recognition or machine translation.

• Evolutionaries: Focus on applying the idea of genomes and DNA in the

evolutionary process to data processing. Evolutionary algorithms will evolve

and adapt to unknown conditions and processes.

• Bayesians: Focus on handling uncertainty using probabilistic inference. Bayesians will take a hypothesis, apply “a priori” thinking believing some

outcomes are more likely, then update a hypothesis as they see more data.

Vision learning and spam filtering are problems tackled by the approach.

• Analogizers: Focus on techniques to match bits of data to each other. Key model is “nearest neighbor” algorithm which can give results to neural network

models.

Machine Learning Technologies

Combine Five Tribes

Domingos, 2015

“One Algorithm to rule them all,

One algorithm to find them,

One Algorithm to bring them all

and in the darkness bind them,

In the Land of Learning where

the Data lies”

Pedro Domingos, 2015

• Advances in the past 30

years have improved

methods to judge “causes”

(or intervention effects) in

non-experimental/

observational settings.

• Applicable to medical and

driving studies – where

difficult or unethical to do

RCT

Finding causality in observational settings

Analytic Methods for

Observational Data

• Propensity score methods

• Marginal structural models

• Use of multiple control

groups

• Instrumental variables

• Regression discontinuity

• Interrupted time series

• Sensitivity Analyses

Computer Vision Object/Lane Detection

Detection Incursion

The Book of Why: The New Science

of Cause and Effect- Pearl and

Mackenzie

The Ladder of Causation, with

representative organisms at each level. Most animals and present-day learning

machines are on the first rung, learning

from association. Tool users, such as

early humans, are on the second rung,

if they act by planning and not merely by imitation. We can also use

experiments to learn the effects of

interventions, and presumably this is

how babies acquire much of their

causal knowledge. On the top rung, counterfactual

learners can imagine worlds that do not

exist and infer reasons for observed

phenomena.

• Simulation allows safe testing of at risk situations and scenarios

• People may act differently in simulation than in the real world because

rewards and benefits of behavior are different

• Differences may be marked in individuals with executive dysfunction.

• Direct evidence of human behavior in naturalistic settings is needed.

• Experience sampling is one way of assessing mental processes and

behavior in the real world.

• Modern technology also allows development of systems combining

accelerometers, GPS, video, other sensors (e.g., of cerebral activity,

eye movement, HR, temperature) to make naturalistic observations of

human movement, physiology and behavior over extended time

frames.

• These systems can assess physiology and behavior from different

vantages (outside looking inside, inside d outside) to evaluate

behavior of people who are seeing, feeling, attending, deciding, erring,

and self-correcting during key activities of daily life.

Conclusions