Authors: Paul Sajda , Adam D. Gerson , Marios G. Philiastides

Single-Trail Analysis of EEG during Rapid Visual Discrimination: Enabling Cortically Coupled Computer Vision

Authors:Paul Sajda, Adam D. Gerson, Marios G. Philiastides

Dept. of Biomedical Engineering, Columbia University, USALucas Parra

Dept. of Biomedical EngineeringCity College of New Work

USA

Thusday, June 16, 2009 1

Outlines

• Introduction• Linear Methods for Single-Trail Analysis• EEG Correlates of Perceptual Decision Making• Identifying Cortical Process leading to

Response Time Variability• EEG-Based Image Triage• Conclusion


Introduction• Linear discrimination of multichannel electroencephalography

(EEG) has used for single-trail detection of neural signatures of visual recognition events.

• Demonstrate: relating neural variability to response variability

• Studies for response accuracy

• Response latency during visual target detection Example: Running in the park with your head phones on,

listening to your favorite tune, & concentrating on your stride, you look up & see a face that you immediately recognize as a high school friend.


Introduction (Con..) She is wearing a hat, glasses & has aged fifteen years since you

last saw her. You & she are running in opposite directions so you only see

her for a fleeting moment, yet you are sure it was her. Your visual system has just effortlessly accomplished a feat that

has thus far baffled best computer vision systems. Such ability for rapid processing of visual information is even

more impressive in light of the fact that neurons are relatively slow processing elements to digital computers, where individual transistors can switch a million times faster than a neuron can spike.


Introduction (Con..) Noninvasive neuroimaging has provided a means to peer into

brain during rapid visual object recognition. Analysis of trial-averaged event-related potentials in EEG has

enabled us to assess speed of visual recognition & discrimination in terms of timing of underlying neural process.

Recent work has used single-trial analysis of EEG to characterize neural activity directly correlated with behavioral variability during tasks involving rapid visual discrimination.

Components extracted from EEG can capture neural correlates of visual recognition & decision-making processes on a trail-by-trail basis.


Introduction (con..)• Noninvasive BCI paradigms: Having a subject consciously modulate brain rhythms Having a subject consciously generate a motor

plan/visual imagery Directly modulating subjects cortical activity by

stimulus frequency (SSVEP) Exploiting specific ERPs (novelty/oddball P300) Focus: Single trail detection of ERPs & relationship to visual

discrimination/recognition

• Cortically coupled computer vision: visual processor performs perception & recognition

• EEG interface detects result (decision).


Linear Methods for Single-Trail Analysis: LDA

• Conventional paradigm of evoked response considers neuronal activity following presentation of a stimulus.

Analyzing EEG activity of multiple electrodes following presentation of an image.

Aim: identify one type of event, visual target recognition, & Differentiate from other

Binary classification based on temporal & spatial profile of potential evoked following stimulus presentation.


Linear Methods for Single-Trail Analysis (Con..)

• EEG activity following each stimulus is recorded as DxT values.

• D: # of channels

• T: # of samples

• Record: 1000 Hz, 64 channels

• ½ sec time window: one word acquire = 32000 samples

• Train classifier: Larger feature vector than N = 100 trials

• Low SNR

• Brute-force classification fail (32000-dim feature vector)



• Exploit prior info: S1: Reduce trial-to-trail variability by filtering (60 Hz ) &

slow drifts (<0.5 Hz), no info carry S2: Reduce dimensionality by grouping signal into L sample

blocks & does not change within time window. S3: Increase # of trails by L redundant samples/window L=50 signal of interest at 10 Hz>> signal: noise Transform original data for each trail with TD dimensions into

L trails: TD/L L=50, N=100, # training exam=5000 train classifier with 640-dimensional feature vector at 10 Hz


Linear Methods for Single-Trail Analysis: LDA (Con..)

• linear classifier: TD/L dimension, good results

• Single training window: L=T, D-dimensional feature vector

• Linear Classification:

• Feature vector x is projected onto an orientation defined by vector w

• Projection, y = wTx, optimally differentiates bet’n two classes.




• Adv: linear voltage combination-immediate

interpretation (current) Coefficients that coupled current with

observed voltages given for linear model by a=<xTy>/<y2>

Angular bracket indicate average over trials & samples.

Coefficients a describe coupling (& correlation) of discrimination component y with sensor activity x.

a & x: D-dim vector(row/column) Strong coupling: low attenuation &

visualized intensity map-sensor projections

Easy to implement Fast Permit real-time operation

• Disadv: does not capture synchronized

activity >10 Hz Neither does it capture activity

that is not a fixed distance in time from stimulus

Only phase-locked activity detection


EEG Correlates of Perceptual Decision Making

• Single-trail LDA to identify cortical correlates of decision-making during rapid discrimination of images.

• Experiment: Psychophysical performance is measured for several subjects during RSVP task

• A series of target (faces) and non-target (cars) trials presented in rapid succession (Fig. 25.2a)

• Simultaneously recording neuronal activity from 64-channel EEG electrode array.

• Stimulus evidence varied with phase coherence (Fig. 25.2b)


EEG Correlates of Perceptual Decision Making (Con..)

• Within trail block: face & car images over a range of phase coherence presented random order.

• 12 face & 12 car, grayscale images (512x512, 8 bits/pixel)

• Equal # of front & side views

• Equal spatial frequency, luminance & contrast

• Subjects are required to discriminate

type of image (face/car) report decision by pressing a

button.



• EEG data acquired in Electrosatically shielded room

• Used: Sensorium EPA-6 Elecrophysiological Amp. from 60 Ag/Agcl scalp electrodes, 3 periocular electrodes placed below left eye, left & right outer canthi

• Sample rate: 1000Hz, 0.01-300 Hz passband, 12 dB/octave HPF & 8th order elliptic LPF.

• Software-based 0.5 Hz HPF (remove dc drifts)

• 60/120 Hz notch filter (minimize line noise) Record: EOG signals, remove motion & blink Motor response & stimulus events recorded on separate

channels



• Identify EEG components that maximally discriminate bet’n 2 conditions

• Identify 2 time window for discrimination

• Identify early (170 ms following stimulus) & late component (>300 ms)

• LD trained by integrating data across both time windows (2D-feature vector)

• Performance improved (higher Az)

• Fig. 25.3: comparison of psychometric & neurometric functions for 1 sub in dataset.

• All subjects a single function can fit behavioral & neuronal datasets & 2 sep functions.



• Data analyzed both stimulus & response-locked conditions

• Both face selective components appear to be more correlated with onset of visual stimulation than response for one subject (Fig. 25.4)

• Discriminate activity significant down to 30% phase coherence (early & late component)

• Randomize Az: significant level of p<0.01.

• Results: neural correlates of perceptual decision making identified using high-spatial density EEG & corresponding component activities




Identifying Cortical Process leading to Response Time Variability

• Significant variability in response time observed across trials in many visual discrimination & recognition.

• Factors: difficulty in discriminating an object on any given trial

• Trial-by-trial variability of subject’s engagement

• Intrinsic variability of neural processing

• Identifying neural activity correlated with response time variability may shed light on the underlying cortical networks responsible

for perceptual decision-making processes & processing latencies that these networks may introduce for a given task


Identifying Cortical Process leading to Response Time Variability (Con..)

Visual target detection: use RSVP & single trial spatial integration of high-density EEG

to identify time course & cortical origins leading to response time variability.

• RSVP paradigm (Fig 25.5)

• Varied scale, pose & position of target objects (people) requires subjects to recognize rather than low-level features

• Continuous sequence of natural scenes

• 4 blocks (50 sequences) rest period: no more than 5 mins.



• Each sequences: 50 images, 50% chance containing one target image with one or more people in a natural scene.

• Target appear: middle 30 images/50 sequences

• Detractor image: remaining natural scene without a person

• Each image: 100 ms

• Fixation cross: 2s bet’n sequences.

• Instructed to press left button of a 3-button mouse with right index finger while fixation cross present, release as soon as recognize target image.

• Used LD: to determine spatial weighting coefficients that optimally discriminate bet’n EEG resulting from different RSVP task conditions (target vs. distractor) over specific temporal window bet’n stimulus & response.



• Integration across sensors enhances signal quality without loss of temporal precision common to trail averaging in ERP.

Resulting discrimination components describe activity specific to target recognition & subsequent response for individual trails.

Intertrial variability estimated by extracting feature from discrimination components.

• Peaks of spatially integrated discriminating components found by fitting a parametric function to extracted component y(t)



• Gaussion Profile: parameterized by its height , width, delay and baseline offset :

• Response-locking of discriminating components determined by computing linear regression coefficients that predict latency of component activity, measured by from response time given by r.

• `j = rj + b [`j-peak latencies, rj-response time for j-th trial, b-offset term for regression] (Fig. 25.6)

• Proportionality factor from response time peak latency regression (): degree of response-locking (%) for each com.

Quantify extent to which component is correlated with response across trails.


0%: pure stimulus 100%: pure response lock=100% : slow responses late activity fast responses early activity=0%: timing activity does not change with response time & therefore stimulus locked.



• Group results: discriminating component activity across 9 participants (Fig. 25.7)

• Scalp projections normalized prior averaging

• A shift of activity from frontal to parietal regions over the course of 200 ms.

• Scaled response times and component peak time are concatenated across subjects

• These registered group response times then projected onto scaled component peak times to estimate degree of response-locking across subjects.

• Group response lock increased from 28% (200ms) to 78% (50 ms) after response.




Significant processing delay introduced by early processing stages

Within 200 ms prior to response (250 ms following stimulus), activity is already, on average, bet’n 25 & 35% response-locked

It is possible that some of this early response-locking may be due to early visual processes (0-250 ms post stimulus

For 9 subjects: correlation Analysis reveal that Discrimination Component activity progressively becomes more response-locked with subsequent Processing stages


EEG-Based Image Triage• EEG system capable of using neural signatures detected

during RSVP to triage sequences of images, reordering them so target images are placed near the beginning of sequence.

• Called: “Cortically coupled computer vision”Robust recognition capabilities of human visual system

(invariance to pose, lighting, scale) use a noninvasive cortical interface (EEG) to intercept

signatures of recognition events- visual processor performs perception & recognition EEG

interface detects the result (decision) of processingThusday, June 16, 2009 28

Image Triage (Con..)• Experimentation: Series of self-paced feedback slides were

presented indicating position of target images within sequence before & after EEG triage.

• Participants completed 2 blocks (50 sequences) with brief rest period lasting no more than 5 minutes bet’n blocks

• During 2nd block: participants were instructed to quickly press left button of 3-button mouse with right index finger as soon as they recognized target images.

• Button press twice if one target immediately followed other.

• Not respond with a button press during 1st block.

• Classify EEG online: used Fisher linear discriminator to estimate a spatial weighting vector that maximally discriminate

between sensor array signals evoked by target & non-target images.


Image Triage (Con..)• 5000 images presented to subject in sequences of 100 images

(with & without motor response)

• Training: 2500 images (50 targets, 2450 non-target)

• Training window: 400-500 ms following stimulus onset was used to extract training data

• Weights updated adaptively with each trail during training

• Classification threshold adjusted to give optimum performance for observed prevalence (Class-prior)

• These weights & threshold were fixed at end of training period & applied to subsequent testing dataset (images: 2501-5000)

• Offline triage: after experiment multiple classifier with different training window onsets were used.


Image Triage (Con..)• Training window: 0~900 ms in steps of 50 ms

• Duration: 50 ms

• After trained, optimal weighting outputs found using logistic regression to discriminate target & non-target images.

• EEG data evoked by 2500 images to train & find inter-classifier weights.

applied to testing dataset evoked by 2nd set 2500 images (2501~5000)

Following experiments: all image sequences were concatenate to create training & testing sequences that each contain 2500 images (target# 50, non-targets# 2450).

Image sequences are restored according to output of classifier with multiple training windows for EEG evoked by every image.


Image Triage (Con..)• Comparison: Sequence triaged based on button response.

• Image restored:

• RT-onset of button response (occurs within1 sec)

• P(target/RT)=0, no response

• Priors P(target) = 0.02

• P(non-target)=0.98

• P(RT/target): Gaussian distributions with mean & variance determined from response times from training sequences

• If target appeared 1st in sequence & 2 button response occurred within 1 sec-assigned to target images

• 2nd response to 2nd target image


Image Triage (Con..)• Testing sequences: if 2 or more within 1 sec, response with

greatest p(target/RT) assigned to image.

• P(RT/non-target) is a mixture of 13 Gaussians (same variance as p(RT/target))

• Means: shifting mean from p(RT/target) 600 ms in past to 700 ms in future, increments of 100 ms, exclude actual mean of p(RT/target)

• Triage results (subject 2): Fig. 25.8

• # of targets as a function of # of distracter images both before & after triage based button press & EEG.

• area under curve generated by plotting fraction of targets as a function of fraction of distracter images presented is used to quantify triage performance.



Image Triage (Con..)

• Area is 0.50 for all unsorted image sequences since target images are randomly distributed throughout sequences.

• Ideal: 1.00

• No significant diff in performance (button-based & EEG-based) triage (0.930.06, 0.92 0.03, p=0.69, N=5)

• EEG (motor & no-motor)(0.920.03, 0.91 0.02, p=0.81, N=5)

• position of target images (black squares) & non-target images (white squares) in images sequences (Raster 25.8b-f)

• Performance (5 subjects): Table 25.1


Image Triage (Con..)

• Performance (5 subjects): Table 25.1

• High-level performance

• Button-based triage performance begins to fail, when subjects do not consistently respond to target stimuli (response time>1 sec), subject 2: only 74%

• EEG (motor & button): increase triage performance


Conclusion

• Invasive & noninvasive EEG recording obtained during RSVP of natural image stimuli have shed on speed, variability & spatiotemporal dynamics of visual processing.

• Future issues: learning/priming/habituation, effect of subject expertise, image type & category

• Application level: intercepting neural correlates of visual discrimination & recognition events that effectively bypass “slow & noisy” motor response loop.



Documents

Authors: Paul Sajda , Adam D. Gerson , Marios G. Philiastides