The relationship between delay period eye movements and visuospatial memory

The relationship between delay period eye movements andvisuospatial memory

Rosanna K Olsen $Rotman Research Institute Baycrest Toronto ON

Canada

Mark Chiew $

Rotman Research Institute Baycrest Toronto ONCanada

Present address Oxford Centre for Functional MRI of theBrain University of Oxford Oxford UK

Bradley R Buchsbaum $


Department of Psychology University of TorontoToronto ON Canada

Jennifer D Ryan $


Department of Psychology and Department ofPsychiatry University of Toronto Toronto ON Canada

We investigated whether overt shifts of attention wereassociated with visuospatial memory performanceParticipants were required to study the locations of a setof visual objects and subsequently detect changes to thespatial location of one of the objects following a briefdelay period Relational information regarding thelocations among all of the objects could be used tosupport performance on the task (Experiment 1) orrelational information was removed during test andlocation manipulation judgments had to be made for asingly presented target item (Experiment 2) Wecomputed the similarity of the fixation patterns in spaceduring the study phase to the fixations made during thedelay period Greater fixation pattern similarity acrossparticipants was associated with higher accuracy whenrelational information was available at test (Experiment1) however this association was not observed when thetarget item was presented in isolation during the testdisplay (Experiment 2) Similarly increased fixationpattern similarity on a given trial (within participants)was associated with successful task performance whenthe relations among studied items could be used forcomparison (Experiment 1) but not when memory forabsolute spatial location was assessed (Experiment 2)This pattern of behavior and performance on the twotasks suggested that eye movements facilitated memoryfor the relationships among objects Shifts of attention

through eye movements may provide a mechanism forthe maintenance of relational visuospatial memory

Introduction

Baddeley (1986) suggested that analogous to artic-ulatory rehearsal of verbal material the oculomotorsystem may play a role in keeping visuospatialinformation lsquolsquoin mindrsquorsquo during a delay period betweenstudy and test This suggestion that eye movementssupport memory echoed similar sentiments expressedby Noton and Stark (1971a b) who theorized that eyemovements are a critical aspect of perception and thesubsequent activation of the corresponding internalimage during recognition Known in the literature aslsquolsquoscanpath theoryrsquorsquo this proposal was supported byreports of striking similarity between the eye movementpatterns observed during memory retrieval and thosewhich occurred during the initial study episode (Fisheramp Monty 1978 Noton amp Stark 1971a b Parker 1978but see Didday amp Arbib 1975 and Walker-SmithGale amp Findlay 1977 for a different interpretation ofthese results) Since then behavioral and neuroscience

Citation Olsen R K Chiew M Buchsbaum B amp Ryan J D (2014) The relationship between delay period eye movementsand visuospatial memory Journal of Vision 14(1)8 1ndash11 httpwwwjournalofvisionorgcontents1418 doi1011671418

Journal of Vision (2014) 14(1)8 1ndash11 1httpwwwjournalofvisionorgcontent1418

doi 10 1167 14 1 8 ISSN 1534-7362 2014 ARVOReceived May 19 2013 published January 8 2014

evidence has indicated that the allocation of spatialattention may support spatial working memory (seeAwh amp Jonides 2001 for a review Awh Jonides ampReuter-Lorenz 1998 Lawrence Myerson amp Abrams2004 Pearson amp Sahraie 2003 Smyth amp Scholey1994) Given that the brain networks responsible forthe planning and execution of eye movements arepartially overlapping with those involved in spatialattention and spatial working memory (Awh Arm-strong amp Moore 2006 Belopolsky amp Theeuwes 2009Hoffman amp Subramaniam 1995 McPeek amp Keller2002) overt shifts of attention during the maintenanceof visuospatial information may benefit memoryperformance Thus investigating the relationship be-tween eye movements and patterns of memory perfor-mance on tasks that tap into different types ofmnemonic processing can help further elucidate thenature of visuospatial memory representations and theneural regions that support them

The relationship between eye movements andmemory has been explored with tasks that require themaintenance andor retrieval of spatial informationFor example during mental imagery and memoryretrieval of spatial information eye fixations weredirected toward locations that were initially studied(Brandt amp Stark 1997 Johansson Holsanova Dew-hurst amp Holmqvist 2012 Johansson amp Johansson2013 Laeng amp Teodorescu 2002) Evidence supportingthe link between eye movements and spatial workingmemory performance specifically has come fromseveral studies that have reported that the disruption ofeye movements during the retention of visuospatialinformation impairs memory performance (Lawrenceet al 2004 Pearson amp Sahraie 2003 Postle 2006)Instructing participants to either maintain fixation ormove their eyes in a task-irrelevant manner likelyprevented both overt andor covert memory rehearsalHowever the task manipulations in these studies couldhave affected memory performance for reasons sec-ondary to the eye movements themselves Disruptingeye movements fundamentally changes the nature ofthe task by imposing additional requirements (egmaintain fixation) and may even increase memorydemands (eg remembering to look at the display in acertain manner) Furthermore while previous studiesdirectly manipulated eye movements through taskinstructions these within subject manipulations couldhave unintentionally affected the way participantsmove their eyes during the lsquolsquofree eye movementrsquorsquoconditions (Godijn amp Theeuwes 2012 Pearson ampSahraie 2003)

Tremblay Saint-Aubin and Jalbert (2006) assessedthe contribution of delay period eye movements tovisuospatial memory performance during the recall ofstudied locations in serial order which requires boththe formation of spatial memory representations as well

as memory for the correct temporal sequence of thestudy item presentations Their findings indicated thateye movements during the delay period to previouslystudied locations benefitted memory performance forserially presented visuospatial information (but seeGodijn amp Theeuwes 2012) but questions remain aboutwhether memory for the spatial andor the temporalinformation was affected by these overt shifts ofattention To address this issue we examined the roleof overt shifts of attention (which occurred naturallywithout the use of explicit task instructions regardingeye movements) for visuospatial memory in twoexperiments and without the additional requirement ofmemory for temporal order

One aspect of short-term visuospatial memoryrehearsal that has been overlooked by previousinvestigations is the relative contribution of eyemovements to the absolute item positions andor thespatial relations among studied items Jiang Olsonand Chun (2000) demonstrated that change detectionfor a tested location was significantly reduced when thesurrounding items were either removed or reconfiguredbetween study and test This evidence suggests thatitem representations are likely not stored entirelyindependently from one another but instead itemswithin a visual scene are encoded with respect to oneanother and the spatial relations among items arerepresented within short-term memory That saidvisuospatial memory performance still remains abovechance when context information is removed thus itseems that both absolute and relative spatial represen-tations are held in short-term memory Thus a majorquestion left unanswered by previous research iswhether overt andor covert shifts of attention servedas a rehearsal mechanism for the absolute spatiallocations or whether delay period shifts of attentionserve to bolster memory for the spatial relationshipsamong studied locations

The present work investigated whether participantsspontaneously rehearsed previously studied locationsduring the delay period of a visuospatial delayed-match-to-sample task in the absence of explicitinstructions regarding eye movements Further weexamined the extent to which eye movements benefittedmemory by strengthening representations for therelative spatial positions among items or whether eyemovements enhanced the absolute spatial representa-tions In two experiments participants viewed a set ofnovel abstract objects while load (number of objectsstudied) and duration of the delay period (time betweenstudy and test displays) were varied in order to achievea wide range of performance on the task A blankscreen was presented during the delay period InExperiment 1 visuospatial memory was probed afterthe delay period by re-presenting all of the studiedobjects in either an intact or manipulated test display

Journal of Vision (2014) 14(1)8 1ndash11 Olsen Chiew Buchsbaum amp Ryan 2

(see Figure 1 upper panel) In manipulated displaysthe position of one of the objects was moved in spacefrom where it was initially studied By contrast the testdisplays in Experiment 2 only contained a single objectwhich was re-presented in either the identical or analtered spatial location Thus in both experimentsmemory for the spatial location of an object wasassessed however participants could use the storedrelational information to guide their decision (Exper-iment 1) or participants were forced to rely moreheavily on memory for the absolute spatial informa-tion as the relational information among items was nolonger available (Experiment 2)

To examine whether overt shifts of attention wereutilized during the delay period to support memoryperformance the spatial distribution of eye movementsduring the delay period was compared to that of thestudy phase using a newly developed similarity metricGreater similarity values reflected greater congruencebetween the spatial distribution of fixations during thestudy and delay periods The relationship between thesimilarity metric and subsequent recognition bothacross and within participants was then assessed foreach experiment The across-participant analysis ex-amined whether the participants who made more delayperiod eye movements back to the studied locationsoverall were also the participants who performed betteron the task the within-participants analysis examined

whether the trials in which participants made moredelay period eye movements back to the studiedlocations were more often the trials on which partic-ipants responded accurately Positive correlations withsimilarity and task performance in both experimentsthat is regardless of whether relative position infor-mation with respect to the other objects is madeavailable would indicate that memory for the absolutespatial locations were maintained and rehearsed via eyemovements By contrast positive correlations withsimilarity and task performance only when informationregarding the relative positions among the objects ispresent in the test display (Experiment 1) but not whenthe target object is presented in isolation (Experiment2) would indicate that the rehearsal of spatial locationsduring the delay period facilitates the formation ofrelational rather than absolute spatial memory repre-sentations More specifically such findings wouldsuggest that the purpose of these eye movements is tointegrate the spatial relations among the studied objectsrather than to encode absolute spatial locations of eachobject (Ryan amp Villate 2009) This work will thenreveal the extent to which eye movements areassociated with successful retention of visuospatialinformation and in particular the specific nature of thevisuospatial information that is associated with delayperiod eye movements

Figure 1 The display sequence of the two visuospatial memory experiments Upper panel Experiment 1 During the study phase

three four or five objects were presented for 2 s followed by a visual mask (500 ms) Following a variable delay period the objects

were displayed in either identical locations (intact trials) or one object was shifted in location as in the example above (manipulated

trials) Lower panel Experiment 2 Study and delay phase were identical to Experiment 1 However during the test phase of

Experiment 2 only the target object was displayed In Experiment 1 participants could rely on either absolute or relative spatial

information to detect changes in spatial position whereas in Experiment 2 participants had to rely predominantly on absolute spatial

information In both examples the red object was shifted to the left by 258 during the test phase compared to its location during the

study phase


Experiment 1

Methods

Participants

Participants were 16 young adults (nine female) aged18ndash32 (M frac14 2225 SDfrac14 423) with normal orcorrected-to-normal vision Participants were recruitedfrom the Rotman Research Institute participant pooland the University of Toronto All participantsprovided informed consent

Apparatus and classification of fixations

Stimuli were presented on a 19-in Dell M991monitor (resolution 1024 middot 768 Dell Round RockTX) Monocular eye movements were recorded with ahead-mounted Eyelink II eye tracker (sample ratefrac14500Hz SR Research Ltd Mississauga Ontario Canada)Eye movement calibration was performed at thebeginning of the experiment and drift correction (58)if needed was performed immediately prior to theonset of each trial Saccades were determined using thebuilt-in EyeLink saccade-detector heuristic accelera-tion and velocity thresholds were set to detect saccadesgreater than 058 of visual angle Blinks are defined asperiods in which the saccade-detector signal wasmissing for three or more samples in a sequenceFixations are defined as the samples remaining after thecategorization of saccades and blinks

Stimuli procedures and design

Participants were presented with a delayed-match-to-sample task in which they decided whether each ofthe objects within the study and test displays matchedin spatial location (Figure 1 upper panel) Eyemovements were recorded throughout the experimentParticipants were not given explicit instructions on howto move their eyes thus they were engaged in freeviewing at all times At the start of each trial a set ofmulticolored abstract objects (three four or fiveobjects) appeared on a gray background for 2 sfollowed by a visual mask for 500 ms The objects werecreated in Corel Draw v 12 (Corel Ottawa OntarioCanada) and scaled so that they were all equivalent insize (visual angle frac14 28) and were brightly colored tominimize perceptual interference The objects wereuniquely designed to minimize resemblance to real-world objects discourage use of associated verballabels to aid in remembering the locations (eg lsquolsquothe catis to the left of the boyrsquorsquo) and were among the set usedin prior work (Olsen Rondina Riggs Meltzer ampRyan 2013 Ryan Leung Turk-Browne amp Hasher2007 Ryan amp Villate 2009) Visual masks were made in

Adobe Photoshop (Adobe Systems Inc San Jose CA)by distorting the study image using a lsquolsquowaversquorsquo functioneach trial had a unique visual mask The resultingmasks were gray screens that contained highly distortedor pixilated versions of the original objects and pixelswere lsquolsquoscatteredrsquorsquo so that they extended beyond theoriginal location of the object The masks were used tominimize visual persistence on the retina so that localluminance changes could not be used to detect a changein an objectrsquos spatial position After the visual mask ablank gray screen was displayed for 770 to 20000 ms(median delay period frac14 5608 ms) Following the delayperiod a prompt was displayed for 500 ms to cue theonset of the test display The test display presented thepreviously studied objects in either the identical spatiallocations (intact display) or one of the objects wasshifted 258 away (either a horizontal or lateral shift)from the studied location (manipulated display) whichresulted in a disruption to the relative and absoluteposition of the object with respect to the otherpresented objects Displays were constructed before theexperiment and consisted of 288 lsquolsquosetsrsquorsquo that werecounterbalanced across participants Within the studyand test displays objects could be presented in 1 of 24possible locations Twelve locations were locatedaround an imaginary circle (similar to a clock) with aradius of 918 and the corresponding lsquolsquolurersquorsquo locationswere shifted 258 laterally or horizontally (either inwardor outward) away from the corresponding originallocation One hundred forty-four trials were presentedin the experiment half of which were intact and half ofwhich were manipulated trials Participants used ahandheld response box to make their test response (ielsquolsquointactrsquorsquolsquolsquomanipulatedrsquorsquo judgments) and were in-structed to respond as quickly and accurately aspossible Participants were not told in advance of thenumber of study objects or the length of the retentioninterval (trial order was randomized) The number ofobjects displayed in each trial (three four or five) wasbalanced equally across the intact and manipulatedtask conditions similarly the length of the delay period(24 different delay periods) was balanced equally acrossthe intact and manipulated conditions as well as thethree load conditions Intact and manipulated displaysas well as the study and test displays were counter-balanced across participants to control for stimulus-specific effects on viewing

Behavioral analysis

Repeated measures ANOVA was used to examinethe effect of test display load and delay length onbehavioral performance (see Supplementary Materials)These factors were also included in a logistic regressionwhich examined the effect of eye movements onbehavioral performance These two analyses produced


a similar set of main and interaction effects thus onlythe results of the logistic regression analysis arereported below

Eye movement analysis

We examined the extent to which the spatialdistribution of delay period fixations mimicked thespatial distribution of fixations that occurred duringthe study phase If participants used overt shifts ofattention to lsquolsquorevisitrsquorsquo studied locations in order to boostvisuospatial memory performance the spatial distri-bution of delay period fixations would be similar tothose during the study phase A similarity metricbetween the two sets of fixation patterns (the set of x-ycoordinates of each fixation and the correspondingfixation duration) was computed using a simplealgorithm developed in Python Fixation patterns wereconverted into a single vector which were collapsedacross fixation order with each entry in the vectorcorresponding to a different spatial location (ie thevector included all possible screen locations) The valueof the entry for each position was proportional to thefixation duration that occurred at that screen locationIn this formulation longer fixation durations at acertain screen location corresponded to larger (higheramplitude) vector values in the corresponding entry ofthe vector To ensure that the similarity metric was notbiased by a disparity in the number of study phase anddelay phase fixations only the first seven fixations fromeach time period were submitted for analysis (onaverage during Experiments 1 and 2 participants

made seven fixations during the study phase and onaverage they made 14 and 11 fixations during the delayphase of Experiments 1 and 2 respectively) Thesefixation lsquolsquomapsrsquorsquo were then spatially smoothed with aGaussian filter (full width at half maximumfrac14 100pixels) to incorporate neighborhood weighting Thatis the amplitude of the vector entries in neighboringpositions to the fixation point scaled with a Gaussiandrop off This ensured that nearby but not exactly co-incident fixations were still granted partial weighting bythe similarity metric Similarity between two fixationmaps was calculated by computing the Euclidean dot-product between the vectors representing each mapLastly the similarity metric was scaled from 0 to 100such that low scores indicated low similarity betweenstudy and delay period fixation patterns and higherscores indicated higher similarity (see Figure 2)

Statistics

Statistical correlations and repeated measures AN-OVA were computed using SPSS (version 200) andlogistic regression analysis was performed in R (version215) using a linear mixed effects model Significantresults were reported for logistic regression coefficientsassociated p values 005 and odds ratios are reportedas an estimate of effect size (Pampel 2000)

Results

Permutation analysis

A permutation analysis was performed in which thedata were initially resampled so that fixation patternsfrom the study phase of a given trial were compared todelay period fixation patterns from a randomly chosentrial The similarity analysis was then performed oneach trial of the shuffled data in order to create a nulldistribution for comparison with the correctly labeleddata Because the object locations varied from trial totrial fixation patterns compared across trials shouldresult in relatively low similarity scores The obtainedsimilarity values were then permuted 1000 times toprovide a null distribution of data to which thecorrectly labeled data could be compared As expectedthis analysis of the permuted data yielded resulted inlow similarity scores (95 CI [836 923])

Similarity analysis results Examining the spatial overlapbetween study-phase and delay-phase fixations

A range of similarity scores was obtained bycomparing the study and delay period fixations for eachtrial and for each subject The mean computedsimilarity value was 1875 (95 CI [1799 1951])critically this mean value was clearly outside of the

Figure 2 Examples of two different trials that yielded either a

relatively high (similarity scorefrac14 723 upper panel) or a

relatively low (similarity score frac14 081 lower panel) similarity

score between the study phase and delay period fixation

patterns Fixations are indicated by the turquoise circles

overlaid onto the task display background Yellow squares

indicate the location of the studied objects (but did not appear

on the screen during the experiment)


95 CI (836 923) of the null distribution asdetermined by the permutation analysis One partici-pantrsquos mean similarity score was identified as an outlierusing Tukeyrsquos (1977) method (based on the interquar-tile range) and this participant was excluded from allfurther analyses

Relationship between similarity scores and accuracy

Participantsrsquo mean similarity scores were correlatedwith their overall task accuracy (percent correct) Asignificant positive relationship between mean similar-ity score and memory performance was found (rfrac14 071ptwo-tailed frac14 0003 Figure 3 left) This significantbetween-subject correlation provided intriguing initialevidence that overt shifts of attention during the delayperiod between study and test may facilitate memoryperformance

Participants who had higher overall similarity scoresalso achieved higher overall accuracy on the task Thissuggests that revisiting studied locations during thedelay period of a particular trial should affect memoryperformance on that same trial To test this hypothesisthe variability of similarity scores across trials withineach participant was examined with respect to testaccuracy A logistic regression analysis was performedthat included task accuracy (correct vs incorrect) as thebinary outcome variable and test display (intact vsmanipulated) delay period length memory load (threefour or five study objects) and similarity score as thepredictor variables

The results from the logistic regression revealed asignificant main effect of similarity score on accuracy(odds ratio 134 p frac14 0038) participants were morelikely to accurately remember the spatial relationsduring trials in which the similarity score was highPerformance was more accurate for intact compared tomanipulated displays (odds ratio 045 p frac14 0029) A

main effect of load (odds ratio 080 pfrac140013) was alsoobserved however the test display and load variablesinteracted in their effect on accuracy (odds ratio 168 p 0001) Accuracy decreased with load for intact trialsand increased with load for manipulated trials (seeSupplementary Figure 1A) The logistic regressionresults also indicated a main effect of delay period onaccuracy (odds ratio 073 p 0001) performancedecreased as time between the study and test phases ofthe task increased (see Supplementary Figure 1B) Inaddition an interaction between the delay period andtest display on accuracy was observed (odds ratio 127pfrac14 0026) due to the fact that the negative correlationbetween delay period duration and task accuracy wasstronger for intact trials (rfrac14044) than formanipulated trials (rfrac14030)

In summary participants who had higher overallsimilarity scores tended to perform more accurately onthe task than participants with lower overall similarityscores Similarly within each participant highersimilarity scores were associated with more accuratememory performance across trials This indicates thatrevisiting previously studied locations during the timebetween study and test benefits either memory for therelations among studied objects andor for the absolutelocations themselves Thus these overt shifts ofattention might serve to strengthen or facilitate theformation of absolute spatial memory representationsor the spatial relations among studied objects

Similarity analysis across the delay period

After establishing that similar study and delay periodfixation patterns resulted in higher memory perfor-mance on the task we next investigated whether thetime in which these maintenance period fixationsoccurred was a factor To address this question a lsquolsquotimebinrsquorsquo analysis was performed in which fixations from

Figure 3 Relationship between overall study-delay fixation similarity scores and task accuracy Similarity score (an index of spatial

overlap between study and delay period fixation patterns) is positively related to overall memory performance when relational

information was present in the test display (Experiment 1 nfrac14 15) but is only weakly related to memory performance when the test

probe was presented in isolation (Experiment 2 n frac14 16) Each point corresponds to a single participant


ten 2-s intervals during the delay phase were indepen-dently compared with the study phase fixations and asimilarity score was computed separately within each ofthe successive delay period time bins (Figure 4) Resultsfrom this analysis showed that similarity was highestearly in the delay period (repeated measures ANOVArevealed a main effect of time bin Ffrac14 444 p 0001)

We next investigated whether including time bin as afactor in the logistic regression improved the ability forsimilarity score to predict accuracy Two linear mixedlogistic regression models were used a reduced modelwhich contained similarity score as a within-subjectfactor (collapsing across all other factors) and a fullmodel which contained both similarity and time bin asfactors (as well as the interaction term) The twomodels were compared with a likelihood ratio test Thefull model was a better predictor of accuracy than thereduced model (v2frac14 403 pfrac14 0047) In addition in thefull model there was a linear time by similarity scoreinteraction zfrac14202 p 0043 The negative z-scoreindicated that the effect of similarity score on accuracydeclines over time To investigate this relationshipfurther logistic regression was performed for each timebin separately to assess the strength of the associationbetween similarity score and accurate task performancethroughout the delay period Results indicted thatduring the first four time bins (0ndash8 s after the studydisplay) there was a significant or marginally significanteffect of similarity score on accuracy (0ndash2 s pfrac14 00762ndash4 s pfrac14 0045 4ndash6 s pfrac14 0033 6ndash8 s pfrac14 0059) Asfor the remaining time bins (which contained fixationsoccurring more than 8 s after the study display)similarity score was not a significant predictor of taskaccuracy (ps 0195) In summary the time binanalysis demonstrated that the spatial patterns of earlydelay period fixations were more similar to those of thestudy phase compared to those of later in the delay

period and that these early delay period fixations weresignificant predictors of task accuracy

Relationship between study period fixations andaccuracy

An additional analysis was conducted on the studyperiod fixations themselves to investigate the specificityof the relationship between delay period eye movementrehearsal patterns and task performance Acrosssubjects no significant correlation was found betweenaccuracy and the mean number of study periodfixations (r frac14 017 p frac14 0545) Similarly a within-subjects logistic regression analysis found no relation-ship between the number of fixations made duringstudy and task accuracy (p frac14 0851) These analysessuggest that task performance was related specificallyto eye movement fixation pattern similarity between thestudy and delay periods and not to the pattern ofviewing enacted during encoding alone

Experiment 2

In Experiment 2 the testing procedure was altered sothat test displays presented only a single target objectwhich precluded the use of the relative spatial relationsamong all of the objects in order to detect a change inspatial position (Bays amp Husain 2008 Jiang et al2000) Therefore successful performance predominantlyrequired participants to remember the absolute spatialposition of each object While participants may haveincidentally encoded andor rehearsed the relativepositions of the studied objects this information wouldnot have been necessary or helpful for task performanceAgain the similarity metric was used to examine theextent to which eye movements during the delay periodmimicked those during the study phase and werepredictive of subsequent memory performance

Methods

Participants

Participants were 16 young adults (12 female) aged19ndash30 (Mfrac14 2138 SDfrac14 270) with normal or corrected-to-normal vision Participants were recruited from theRotman Research Institute participant pool and theUniversity of Toronto and provided informed consent

Apparatus

Apparatus and eye movement recording procedureswere identical to Experiment 1

Figure 4 Similarity scores plotted separately for the ten 2-s time

bins demonstrating how fixation similarity patterns change as a

function of time elapsed since study Error bars depict SE


Procedure

The task used in Experiment 2 was identical to thatused in Experiment 1 with the following exceptionDuring the test phase only one object was presentedbut as in Experiment 1 this target object was presentedin either the same spatial location in which it wasviewed during the study phase or in a shifted spatiallocation (Figure 1 lower panel) Note that the targetobject was moved exactly the same distance (on average258 in visual angle) as in Experiment 1

Results

Similarity scores and the null distribution werecalculated in the same manner as in Experiment 1 Themean similarity value obtained in Experiment 2 was2399 95 CI (2322 2476) and was well outside the95 CI of the null distribution computed on thepermuted trials (1104 1208) In contrast to Experi-ment 1 the relationship between study-delay fixationpattern similarity and task performance was notstatistically significant (Figure 3 right r frac14 043ptwo-tailed frac14 0096) Revisiting studied interest regionsduring the delay period did not have the same beneficialeffect on memory for absolute location information asit did when both absolute and relational informationwas available as tested in Experiment 1

Logistic regression analysis was again performed inwhich task accuracy was the binary outcome variableand test display (intact vs manipulated) delay periodlength memory load (three four or five objects) andsimilarity score were the predictor variables Thesimilarity between study period and delay periodfixations was not associated with increased memoryperformance (pfrac14 0799) The main effect of condition(pfrac14 0327) and the main effect of load (pfrac14 0139) werenot significant As in Experiment 1 accuracy decreasedwith load for intact trials and increased with load formanipulated trials resulting in a significant test displayby load variable interaction (odds ratio 121 pfrac14 003Supplementary Figure 1C) As in Experiment 1 delaylength was significantly related to accuracy (odds ratio068 p 0001) performance decreased as timebetween the study and test phases of the task increased(Supplementary Figure 1D) Finally a test display bydelay variable interaction was observed (odds ratio176 p 0001) This interaction was driven by thesignificant negative relationship between delay lengthand accuracy on intact trials (rfrac14063 pfrac14 0001)which was not present during manipulated trials (r frac14031 p frac14 0140)

Although high similarity between fixation patterns atthe study and delay period was observed this within-subjects analysis demonstrated that in contrast to thefindings from Experiment 1 revisiting the studied

locations during the delay period was not beneficial fortask performance All together these results suggestthat eye movements made to previously occupiedlocations may help form andor strengthen memory forspatial relations among the objects in memory but thatshifts of eye movements do not strengthen the memoryfor the absolute position of each object location

Discussion

The present results provide intriguing new evidencefor the role of overt shifts of attention duringvisuospatial memory maintenance While the func-tional role of eye movements during memory mainte-nance has previously been explored to the best of ourknowledge this is the first study that specificallyprobed the nature of the memory representations(absolute vs relational) that are affected by spontane-ously produced eye movements Relational spatialmemory was superior for participants who reinstatedfixation patterns during the delay period that mimickedthe fixation patterns enacted during the study phase Inthe same manner when fixation patterns from thestudy and delay phases were compared within partic-ipants greater fixation pattern similarity was associatedwith higher memory performance We propose thatshifts of eye movementsmdashie overt shifts of atten-tionmdashwere used to actively build and maintainvisuospatial information from the study phase so that itcould be accurately compared to the test probeCritically while overt shifts of attention were sponta-neously executed during both experiments eye move-ments during the delay period were not associated withsubsequent memory for the absolute spatial locationsbut they were related to more accurate memory forspatial relations

An alternate interpretation of the current findings isthat the eye movement patterns reported here simplyreflect better encoding of the object locations during thestudy phase of the trial In other words it is possible thatparticipants who more effectively encoded the objectlocations initially were more likely to revisit those samelocations later and therefore more effective encodinginstead of rehearsal per se benefitted subsequentmemory While we cannot definitively rule out thisexplanation we believe that this lsquolsquoepiphenomenalrsquorsquoaccount of the data is not likely to be accurate for tworeasons First fixation patterns at study were notpredictive of performance at test Previous work fromour group has demonstrated that the number offixations made during the encoding of face stimuli wassignificantly related to subsequent memory (ChanKamino Binns amp Ryan 2011) However in the currentstudy neither the number of fixations made to each of


the studied objects nor the amount of time spentfixating on the studied objects was related to memoryperformance While the current experiment was notdesigned to test for this dissociation this suggests thatthe number of fixations made during encoding maysupport memory for items as in our prior work but notfor spatial relations as tested here Secondly similarityscores were higher overall in Experiment 2 than inExperiment 1 but accuracy was worse and similaritypatterns were not correlated with task accuracy Insummary eye movements during study were not relatedto subsequent memory performance overall but insteaddelay period eye movements were significantly related tobetter memory for spatial relations

Influenced by Hebbrsquos work in 1968 Noton and Stark(1971a b) provided early evidence that a personrsquos eyemovements might follow a unique lsquolsquoscanpathrsquorsquo for aparticular image during study and subsequent retrievalof that image (see Foulsham amp Kingstone 2013 for arecent evaluation of scanpath theory) In a later studyBrandt and Stark (1997) suggested lsquolsquo that eyemovements during imagery reflect the mental process ofactivating and arranging the part images of a complexscene into their proper locationsrsquorsquo (p 33) Scanpaththeory would make similar predictions as those testedhere The extent to which study and delay periodfixations spatially overlap would facilitate memoryretrieval and predict subsequent memory performanceThe current investigation however did not examinethe temporal order of fixations that scanpath theoryalso suggests is an important component for reactiva-tion of the stored memory trace Furthermore it is stillunclear whether the eye movement records themselvesbecome part and parcel of the stored memoryrepresentation as proposed by Stark and colleaguesThus future investigations will be needed to fully testwhether the current data support the various predic-tions of scanpath theory

The present results may also inform the recently re-energized debate of lsquolsquolooking at nothingrsquorsquo (FerreiraApel amp Henderson 2008 Richardson AltmannSpivey amp Hoover 2009)mdasha proposal based on findingsthat participants often make eye movements topreviously studied but now empty locations whenmemories for information (eg visual objects orsemantic facts simultaneously presented in the auditorymodality) associated with those locations are probed(Hoover amp Richardson 2008 Johansson amp Johansson2013 Richardson amp Spivey 2000 Spivey amp Geng2001) Johansson and Johansson (2013) investigatedthe role of eye movements in facilitating the recall ofspatial relationsmdashand critically they found that gazeposition during memory retrieval facilitated the abilityto recall spatial relationships among studied objects(but not memory for the objects themselves) Interest-ingly in the studies by Richardson and Spivey (2000)

participants made eye fixations back to the studiedlocations even when location memory itself was notprobed nor was accurate performance contingent uponsuccessful retention of location information or object-location associations however such behavior was notassociated with memory performance Likewise par-ticipants in Experiment 2 of the present study exhibitedrehearsal of previously studied locations in their eyemovement patterns even when it was not beneficial fortask performance Taken together the current findingsalong with those that investigated the role of eyemovements during memory recall suggest that eyemovements play a special role in accessing andormaintaining spatial relationships among study objectsbut these eye movements do not confer benefits for theproperties of objects or facts that were associated withspecific spatial locations

The repeated sampling of studied regions in Exper-iment 1 likely involves the extended engagement ofneural regions that are involved both in the processingof spatial relations and in the modulation of visuo-spatial attention It has been traditionally thought thatthe parietal lobes support processing of both relationaland absolute spatial information and that the left andright hemispheres differentially contribute to these twotypes of spatial relations (Jager amp Postma 2003)However a recent neuroimaging investigation foundthat relational representations rely more on parietalregions while absolute representations involve thehippocampus during spatial navigation (BaumannChan amp Mattingley 2012) These parietal lobe regionsalso may overlap with areas known to be involved invisuospatial attention and furthermore may be func-tionally coupled with neural regions responsible foroculomotor planning such as the frontal eye fields andthe superior colliculus (Belopolsky amp Theeuwes 2009)Thus delay period shifts of attention may serve toreactivate the neural circuitry involved in eye move-ments (frontal eye fields) visuospatial attention (fron-tal-parietal network) spatial processing and memory(parietal and hippocampal circuits) This reactivationcould result in stronger relational spatial representa-tions and consequently more accurate performancewhen this maintained representation is eventuallycompared to the test probe

While the present findings suggest that spontaneousovert shifts of attention may serve as a visuospatialrehearsal mechanism this interpretation does not ruleout the role of covert shifts of attention in rehearsal Forexample Godijn and Theeuwes (2012) recently providedevidence that overt shifts of attention were no betterthan covert shifts in memory for serially presentedspatial locations Thus taken together with thesefindings it is likely that the rehearsal system does notrely solely on the execution of oculomotor actions butcan be additionally subserved through covert shifts of


attention a process that is supported by a similar neuralarchitecture Indeed electrophysiological recordings innonhuman primates have revealed that the frontal eyefields are involved in the allocation and maintenance ofspatial information even in the absence of overt eyemovements (Armstrong Chang amp Moore 2009) Thusfurther investigation using cognitive neurosciencemethodology will lead to a better understanding of therelative roles of overt and covert shifts of attention inmemory for absolute and relative spatial locations

Conclusions

In summary overt shifts of attention were associatedwith superior relational spatial memory performanceThese findings have important implications for betterunderstanding memory impairments observed in con-ditions such as Alzheimerrsquos disease in which eyemovements are also affected (Crutcher et al 2009Daffner Scinto Weintraub Guinessey amp Mesulam1992) Given the association between eye movementsand successful memory performance as observed herefuture work could focus on the utilization of eyemovement patterns for early detection of memorydecline and for evaluating the efficacy of cognitiverehabilitation programs

Keywords scanning saccades binding short-termmemory delayed-match-to-sample working memory

Acknowledgments

We would like to thank Rida Anmol Maria AyalaJana Kube and Lingqian Li for assistance with datacollection This research was supported in part by aTier II Canada Research Chair Award and a CanadianInstitutes of Health Research Operating Grant award-ed to Jennifer D Ryan

Commercial relationships noneCorresponding author Rosanna K OlsenEmail rolsenresearchbaycrestorgAddress Rotman Research Institute Baycrest Tor-onto ON Canada

References

Armstrong K M Chang M H amp Moore T (2009)Selection and maintenance of spatial information byfrontal eye field neurons Journal of Neuroscience29(50) 15621ndash15629

Awh E Armstrong K M amp Moore T (2006) Visualand oculomotor selection Links causes and impli-cations for spatial attention Trends in CognitiveSciences 10(3) 124ndash130

Awh E amp Jonides J (2001) Overlapping mechanismsof attention and spatial working memory Trends inCognitive Sciences 5(3) 119ndash126

Awh E Jonides J amp Reuter-Lorenz P A (1998)Rehearsal in spatial working memory Journal ofExperimental Psychology Human Perception ampPerformance 24(3) 780ndash790

Baddeley A D (1986) Working memory Oxford UKOxford University Press

Baumann O Chan E amp Mattingley J B (2012)Distinct neural networks underlie encoding ofcategorical versus coordinate spatial relations duringactive navigation NeuroImage 60(3) 1630ndash1637

Bays P M amp Husain M (2008) Dynamic shifts oflimited working memory resources in human visionScience 321(5890) 851ndash854

Belopolsky A V amp Theeuwes J (2009) When areattention and saccade preparation dissociatedPsychological Science 20(11) 1340ndash1347

Brandt S amp Stark L (1997) Spontaneous eyemovements during visual imagery reflect the contentof the visual scene Journal of Cognitive Neurosci-ence 9(1953) 27ndash38 Retrieved from httpwwwmitpressjournalsorgdoiabs101162jocn19979127

Chan J P K Kamino D Binns M A amp Ryan J D(2011) Can changes in eye movement scanning alterthe age-related deficit in recognition memoryFrontiers in Psychology 2(May) 1ndash11

Crutcher M D Calhoun-Haney R Manzanares CM Lah J J Levey A I amp Zola S M (2009)Eye tracking during a visual paired comparison taskas a predictor of early dementia American Journal ofAlzheimerrsquos Disease amp Other Dementias 24(3) 258ndash266

Daffner K R Scinto L F M Weintraub SGuinessey J E amp Mesulam M M (1992)Diminished curiosity in patients with probableAlzheimerrsquos disease as measured by exploratory eyemovements Neurology 42(2) 320ndash328

Didday R L amp Arbib M A (1975) Eye movementsand visual perception A lsquolsquotwo visual systemrsquorsquo modelInternational Journal of Man-Machine Studies 7(4)547ndash569 doi101016S0020-7373(75)80032-0

Ferreira F Apel J amp Henderson J M (2008) Takinga new look at looking at nothing Trends inCognitive Sciences 12(11) 405ndash410 doi101016jtics200807007

Fisher D F amp Monty R A (1978) Visual recognitionmemory for binary pictures Another look Journalof Experimental Psychology Human Learning ampMemory 4(2) 158ndash164


Foulsham T amp Kingstone A (2013) Fixation-dependent memory for natural scenes An experi-mental test of scanpath theory Journal of Experi-mental Psychology General 142(1) 41ndash56 doi101037a0028227

Godijn R amp Theeuwes J (2012) Overt is no betterthan covert when rehearsing visuo-spatial informa-tion in working memory Memory amp Cognition40(1) 52ndash61

Hebb D O (1968) Concerning imagery PsychologicalReview 75(6) 466ndash477

Hoffman J E amp Subramaniam B (1995) The role ofvisual attention in saccadic eye movements Per-ception amp Psychophysics 57(6) 787ndash795

Hoover M A amp Richardson D C (2008) When factsgo down the rabbit hole contrasting features andobjecthood as indexes to memory Cognition 108(2)533ndash542 doi101016jcognition200802011

Jager G amp Postma A (2003) On the hemisphericspecialization for categorical and coordinate spatialrelations A review of the current evidence Neuro-psychologia 41(4) 504ndash515

Jiang Y Olson I R amp Chun M M (2000)Organization of visual short-term memory Journalof Experimental Psychology Learning Memory ampCognition 26(3) 683ndash702

Johansson R Holsanova J Dewhurst R ampHolmqvist K (2012) Eye movements during scenerecollection have a functional role but they are notreinstatements of those produced during encodingJournal of Experimental Psychology Human Per-ception amp Performance 38(5) 1289ndash1314 doi101037a0026585

Johansson R amp Johansson M (2013) Look here eyemovements play a functional role in memoryretrieval Psychological Science (E-pub ahead ofprint) doi1011770956797613498260

Laeng B amp Teodorescu D (2002) Eye scanpathsduring visual imagery reenact those of perception ofthe same visual scene Cognitive Science 26 207ndash231

Lawrence B M Myerson J amp Abrams R A (2004)Interference with spatial working memory An eyemovement is more than a shift of attentionPsychonomic Bulletin amp Review 11(3) 488ndash494

McPeek R M amp Keller E L (2002) Superiorcolliculus activity related to concurrent processing ofsaccade goals in a visual search task Journal ofNeurophysiology 87 1805ndash1815

Noton D amp Stark L (1971a) Scanpaths in eyemovements during pattern perception Science171(3968) 308ndash311

Noton D amp Stark L (1971b) Scanpaths in saccadiceye movements while viewing and recognizingpatterns Vision Research 11(9) 929ndash942

Olsen R K Rondina Ii R Riggs L Meltzer J A ampRyan J D (2013) Hippocampal and neocorticaloscillatory contributions to visuospatial binding andcomparison Journal of Experimental PsychologyGeneral 142(4) 1335ndash1345 doi101037a0034043

Pampel F C (2000) Logistic regression A primerThousand Oaks CA Sage

Parker R E (1978) Picture processing during recog-nition Journal of Experimental Psychology HumanPerception amp Performance 4(2) 284ndash93

Pearson D G amp Sahraie A (2003) Oculomotorcontrol and the maintenance of spatially andtemporally distributed events in visuo-spatial work-ing memory Quarterly Journal of ExperimentalPsychology Section A Human Experimental Psy-chology 56A(7) 1089ndash1111

Postle B R (2006) Working memory as an emergentproperty of the mind and brain Neuroscience139(1) 23ndash38

Richardson D C Altmann G T M Spivey M J ampHoover M A (2009) Much ado about eyemovements to nothing A response to Ferreira et alTaking a new look at looking at nothing Trends inCognitive Sciences 13(6) 235ndash236 doi101016jtics200902006

Richardson D C amp Spivey M J (2000) Represen-tation space and Hollywood Squares Looking atthings that arenrsquot there anymore 76 269ndash295

Ryan J D Leung G Turk-Browne N B amp HasherL (2007) Assessment of age-related changes ininhibition and binding using eye movement moni-toring Psychology and aging 22(2) 239ndash250 doi1010370882-7974222239

Ryan J D amp Villate C (2009) Building visualrepresentations The binding of relative spatialrelations across time Visual Cognition 17(1ndash2) 254ndash272

Smyth M M amp Scholey K A (1994) Interference inimmediate spatial memory Memory amp Cognition22(1) 1ndash13

Spivey M J amp Geng J J (2001) Oculomotormechanisms triggered by imagery and memory Eyemovements to absent objects Psychological Re-search 65(4) 235ndash241

Tremblay S Saint-Aubin J amp Jalbert A (2006)Rehearsal in serial memory for visual-spatial infor-mation Evidence from eye movements Psycho-nomic Bulletin amp Review 13(3) 452ndash457

Tukey J W (1977) Exploratory data analysis ReadingMA Addison-Wesley Publishing Company

Walker-Smith G Gale A amp Findlay J (1977) Eyemovement strategies involved in face perceptionPerception 6 313ndash326 Retrieved from httpwwwperceptionwebcomfulltextp06p060313pdf


evidence has indicated that the allocation of spatialattention may support spatial working memory (seeAwh amp Jonides 2001 for a review Awh Jonides ampReuter-Lorenz 1998 Lawrence Myerson amp Abrams2004 Pearson amp Sahraie 2003 Smyth amp Scholey1994) Given that the brain networks responsible forthe planning and execution of eye movements arepartially overlapping with those involved in spatialattention and spatial working memory (Awh Arm-strong amp Moore 2006 Belopolsky amp Theeuwes 2009Hoffman amp Subramaniam 1995 McPeek amp Keller2002) overt shifts of attention during the maintenanceof visuospatial information may benefit memoryperformance Thus investigating the relationship be-tween eye movements and patterns of memory perfor-mance on tasks that tap into different types ofmnemonic processing can help further elucidate thenature of visuospatial memory representations and theneural regions that support them

The relationship between eye movements andmemory has been explored with tasks that require themaintenance andor retrieval of spatial informationFor example during mental imagery and memoryretrieval of spatial information eye fixations weredirected toward locations that were initially studied(Brandt amp Stark 1997 Johansson Holsanova Dew-hurst amp Holmqvist 2012 Johansson amp Johansson2013 Laeng amp Teodorescu 2002) Evidence supportingthe link between eye movements and spatial workingmemory performance specifically has come fromseveral studies that have reported that the disruption ofeye movements during the retention of visuospatialinformation impairs memory performance (Lawrenceet al 2004 Pearson amp Sahraie 2003 Postle 2006)Instructing participants to either maintain fixation ormove their eyes in a task-irrelevant manner likelyprevented both overt andor covert memory rehearsalHowever the task manipulations in these studies couldhave affected memory performance for reasons sec-ondary to the eye movements themselves Disruptingeye movements fundamentally changes the nature ofthe task by imposing additional requirements (egmaintain fixation) and may even increase memorydemands (eg remembering to look at the display in acertain manner) Furthermore while previous studiesdirectly manipulated eye movements through taskinstructions these within subject manipulations couldhave unintentionally affected the way participantsmove their eyes during the lsquolsquofree eye movementrsquorsquoconditions (Godijn amp Theeuwes 2012 Pearson ampSahraie 2003)

Tremblay Saint-Aubin and Jalbert (2006) assessedthe contribution of delay period eye movements tovisuospatial memory performance during the recall ofstudied locations in serial order which requires boththe formation of spatial memory representations as well

as memory for the correct temporal sequence of thestudy item presentations Their findings indicated thateye movements during the delay period to previouslystudied locations benefitted memory performance forserially presented visuospatial information (but seeGodijn amp Theeuwes 2012) but questions remain aboutwhether memory for the spatial andor the temporalinformation was affected by these overt shifts ofattention To address this issue we examined the roleof overt shifts of attention (which occurred naturallywithout the use of explicit task instructions regardingeye movements) for visuospatial memory in twoexperiments and without the additional requirement ofmemory for temporal order

One aspect of short-term visuospatial memoryrehearsal that has been overlooked by previousinvestigations is the relative contribution of eyemovements to the absolute item positions andor thespatial relations among studied items Jiang Olsonand Chun (2000) demonstrated that change detectionfor a tested location was significantly reduced when thesurrounding items were either removed or reconfiguredbetween study and test This evidence suggests thatitem representations are likely not stored entirelyindependently from one another but instead itemswithin a visual scene are encoded with respect to oneanother and the spatial relations among items arerepresented within short-term memory That saidvisuospatial memory performance still remains abovechance when context information is removed thus itseems that both absolute and relative spatial represen-tations are held in short-term memory Thus a majorquestion left unanswered by previous research iswhether overt andor covert shifts of attention servedas a rehearsal mechanism for the absolute spatiallocations or whether delay period shifts of attentionserve to bolster memory for the spatial relationshipsamong studied locations

The present work investigated whether participantsspontaneously rehearsed previously studied locationsduring the delay period of a visuospatial delayed-match-to-sample task in the absence of explicitinstructions regarding eye movements Further weexamined the extent to which eye movements benefittedmemory by strengthening representations for therelative spatial positions among items or whether eyemovements enhanced the absolute spatial representa-tions In two experiments participants viewed a set ofnovel abstract objects while load (number of objectsstudied) and duration of the delay period (time betweenstudy and test displays) were varied in order to achievea wide range of performance on the task A blankscreen was presented during the delay period InExperiment 1 visuospatial memory was probed afterthe delay period by re-presenting all of the studiedobjects in either an intact or manipulated test display












study phase


Experiment 1

Methods

Participants







Behavioral analysis







Statistics


Results

































Experiment 2


Methods

Participants


Apparatus






Procedure


Results





Discussion












Conclusions



Acknowledgments



References
























































study phase


Experiment 1

Methods

Participants







Behavioral analysis







Statistics


Results

































Experiment 2


Methods

Participants


Apparatus






Procedure


Results





Discussion












Conclusions



Acknowledgments



References














































Experiment 1

Methods

Participants







Behavioral analysis







Statistics


Results

































Experiment 2


Methods

Participants


Apparatus






Procedure


Results





Discussion












Conclusions



Acknowledgments



References


















































Statistics


Results

































Experiment 2


Methods

Participants


Apparatus






Procedure


Results





Discussion












Conclusions



Acknowledgments



References

































































Experiment 2


Methods

Participants


Apparatus






Procedure


Results





Discussion












Conclusions



Acknowledgments



References



















































Experiment 2


Methods

Participants


Apparatus






Procedure


Results





Discussion












Conclusions



Acknowledgments



References














































Procedure


Results





Discussion












Conclusions



Acknowledgments



References






















































Conclusions



Acknowledgments



References















































Conclusions



Acknowledgments



References











































































Documents

The relationship between delay period eye movements and visuospatial memory