Ludwig, C. J. H. , Davies, J. R., & Eckstein, MP. (2014 ... · Casimir J.H. Ludwig and J. Rhys Davies University of Bristol, School of Experimental Psychology, Bristol, UK Miguel

Ludwig, C. J. H., Davies, J. R., & Eckstein, MP. (2014). Fovealanalysis and peripheral selection during active visual sampling.Proceedings of the National Academy of Sciences of the UnitedStates of America, 111(2), E291-E299.https://doi.org/10.1073/pnas.1313553111

Peer reviewed version

Link to published version (if available):10.1073/pnas.1313553111

Link to publication record in Explore Bristol ResearchPDF-document

This is the author accepted manuscript (AAM). The final published version (version of record) is available onlinevia PNAS at http://www.pnas.org/content/111/2/E291. Please refer to any applicable terms of use of thepublisher.

University of Bristol - Explore Bristol ResearchGeneral rights

This document is made available in accordance with publisher policies. Please cite only thepublished version using the reference above. Full terms of use are available:http://www.bristol.ac.uk/pure/user-guides/explore-bristol-research/ebr-terms/

https://doi.org/10.1073/pnas.1313553111

https://doi.org/10.1073/pnas.1313553111

https://research-information.bris.ac.uk/en/publications/foveal-analysis-and-peripheral-selection-during-active-visual-sampling(b1f1a835-fabf-456c-9f54-469b9bef4202).html

https://research-information.bris.ac.uk/en/publications/foveal-analysis-and-peripheral-selection-during-active-visual-sampling(b1f1a835-fabf-456c-9f54-469b9bef4202).html

Foveal analysis and peripheral selection during active visualsampling

Casimir J.H. Ludwig and J. Rhys DaviesUniversity of Bristol, School of Experimental

Psychology, Bristol, UK

Miguel P. EcksteinUniversity of California Santa Barbara,

Department of Psychological and Brain Sciences,Santa Barbara, United States

Human vision is an active process in which information is sampled duringbrief periods of stable fixation in between gaze shifts. Foveal analysis servesto identify the currently fixated object and has to be coordinated with a periph-eral selection process of the next fixation location. Models of visual searchand scene perception typically focus on the latter, without considering fovealprocessing requirements. We developed a dual-task noise classification tech-nique that enables identification of the information uptake for foveal analy-sis and peripheral selection within a single fixation. Human observers hadto use foveal vision to extract visual feature information (orientation) fromdifferent locations for a psychophysical comparison. The selection of to-be-fixated locations was guided by a different feature (luminance contrast). Weinserted noise in both visual features and identified the uptake of informationby looking at correlations between the noise at different points in time and be-haviour. Our data show that foveal analysis and peripheral selection proceededcompletely in parallel. Peripheral processing stopped some time before theonset of an eye movement, but foveal analysis continued during this period.Variations in the difficulty of foveal processing did not influence the uptake ofperipheral information and the efficacy of peripheral selection, suggesting thatfoveal analysis and peripheral selection operated independently. These resultsprovide important theoretical constraints on how to model target selection inconjunction with foveal object identification: in parallel and independently.

Keywords: Human vision, Eye movement control, Attention, Classificationimage, Dual-task

Significance Statement

Picking up visual information from our envi-ronment in a timely manner is the starting pointof adaptive visual-motor behaviour. Humans andother animals with foveated visual systems extractvisual information through a cycle of brief fixa-tions interspersed with gaze shifts. Object identi-fication typically requires foveal analysis (limitedto a small region of central vision). In addition,

the next fixation location needs to be selected us-ing peripheral vision. How does the brain coor-dinate these two tasks on the short time scale ofindividual fixations? We have developed a novelbehavioural methodology that allows us to identifythe coordination between foveal and peripheral in-formation processing. The uptake of informationfor foveal analysis and peripheral selection occurscompletely in parallel and independently.

2 CASIMIR J.H. LUDWIG AND J. RHYS DAVIES

Introduction

Almost all human visually guided behaviour re-lies on the selective uptake of information, dueto sensory and cognitive limitations. On the sen-sory side, the sampling of visual input by theretinal mosaic of photoreceptors becomes increas-ingly sparse and irregular away from central vi-sion [1]. In addition, fewer cortical neurons aredevoted to the analysis of peripheral visual infor-mation (cortical magnification; [2, 3]). Humansand other animals with so-called foveated visualsystems have evolved gaze shifting mechanisms toovercome these limitations. Saccadic eye move-ments serve to rapidly and efficiently deploy gazeto objects and regions of interest in the visual field.Sampling the environment appropriately with gazeis the starting point of adaptive visual-motor be-haviour [4, 5].

Studies have shown that saccadic eye move-ments are guided by analysis of information inthe visual periphery up to 80-100 ms prior to sac-cade execution [6, 7, 8]. However, active visiontypically requires humans not only also to anal-yse information in the visual periphery to decidewhere to fixate next (peripheral selection), but alsoto analyse the information at the current fixationlocation (foveal analysis). Not much is knownabout how foveal analysis and peripheral selec-tion are coordinated and interact. In this regard,we need to know (i) whether and to what extentfoveal analysis and peripheral selection are con-strained by a common bottleneck or limited capac-ity resource,and (ii) how time within a fixation isallocated to these two tasks.

CL was supported by grants from the UK Engineeringand Physical Sciences Research Council:EP/E054323/1 (Advanced Research Fellowship) andEP/I032622/1. ME received support from NSF0819592.

Capacity limitations are ubiquitous in human vi-sual processing. There is a long-standing debate onthe extent to which visual attention may be focusedon different locations in the visual field [9, 10, 11].If foveal analysis and peripheral selection both re-quire a spatial attentional “spotlight”, the coordi-nation of these two tasks will be constrained byway in which this spotlight can be configured. Forexample, the size of the spotlight may vary withthe processing difficulty of foveal information, asin tunnel vision [12, 13]. Similarly, in both reading[14] and scene viewing [15] a reduction in the per-ceptual span has been reported with higher fovealload. A high foveal processing load can also pre-vent distraction from irrelevant visual informationin the periphery [16]. These findings suggest thatthere may be interactions between foveal analysisand peripheral selection [17, 18], in that the gainon peripheral information processing may vary ac-cording to the foveal processing load.

A useful way to think of the coordination be-tween foveal analysis and peripheral selection isto picture the temporal profile of information ex-traction over the course of a fixation period. Fig-ure 1 shows some schematic profiles, or integra-tion windows, for foveal and peripheral informa-tion (shown in black and grey respectively). PanelsA – C chart the progression in the extent to whichthe extraction of peripheral visual information iscontingent upon the completion of foveal analysis:from completely contingent (A – serial), troughpartly contingent (B – cascaded), to completelyparallel (C). This temporal relation between fovealanalysis and peripheral selection is a core assump-tion of models of eye movement control in reading[19, 14, 20, 21] and other visual-motor domains[22, 23]. Finally, panel D demonstrates a hypothet-ical trade-off between an increase in foveal loadand a decreased peripheral gain. In this example,the foveal integration window is extended to re-flect the higher processing load. The duration ofthe peripheral window is also extended, but by asmaller amount and its amplitude is reduced. Notethat the accuracy of peripheral selection will be de-

INSERT SHORTTITLE COMMAND IN PREAMBLE 3

termined by both the amplitude and the duration ofthe integration window.

0 100 200 300 400

Serial

wei

ght

0 100 200 300 400

Cascaded

0 100 200 300 400

Parallel

time (ms)

wei

ght

0 100 200 300 400

Parallel (increased foveal load)

time (ms)

fovealperipheral

A B

C D

Figure 1. Hypothetical temporal weighting func-tions for foveal analysis and peripheral selection.A – Strict serial model: peripheral information isonly analysed once foveal processing is complete.B – A weaker version of the serial model in whichperipheral information is processed once some cri-terion amount of foveal analysis is complete. C –Parallel model in which foveal analysis and periph-eral selection start together. In A-C the time win-dow for peripheral selection is shorter than that forfoveal analysis, reflecting the primary importanceof the latter. D – Manipulation of foveal load. Asfoveal processing difficulty is increased, more timeis taken to analyse the foveal information. Thetime window for peripheral selection extends aswell, but by a smaller amount. In addition, thegain of peripheral processing is lower, resulting inattenuation of the amplitude of the weighting func-tion.

A potentially powerful way to identify the co-ordination between foveal analysis and peripheralselection is then to estimate these underlying inte-gration windows directly, under conditions of vari-able foveal processing load. Identifying these win-dows is far from trivial: it involves determiningwhat information is being processed, from where,and at what point in time during an individual fixa-

tion. We have developed a dual-task noise classifi-cation approach [24, 25, 26] that allows us to iden-tify what information is used by the observer forwhat “task” over the brief time scale of a single fix-ation. Using this method, we show that the uptakeof information for foveal analysis and peripheralselection proceeds independently and in parallel.

Results

Our experimental paradigm is illustrated in Fig-ure 2. It aims to mimic the demands of activevisual sampling. The observer has to analyse in-formation at multiple locations using foveal vision(foveal analysis), utilise peripheral information toguide a saccade (peripheral selection) and use thecollected information to make some overall per-ceptual decision about the state of the visual world.Critically, foveal analysis and peripheral selectionare based on different visual dimensions (tilt andcontrast respectively). We insert temporal noise inboth visual dimensions and relate this noise to thebehavioural outcomes.

Eight human observers took part in a compara-tive tilt judgement task. An initially fixated pat-tern (which we refer to as the ‘foveal target’)was tilted away from vertical by a small amount(1◦ or 2◦) in either a clockwise or anti-clockwisedirection. Three peripheral patterns were pre-sented, one of which (‘peripheral target’) had aslightly higher luminance contrast than the othertwo (‘non-targets’). All three patterns were inde-pendently tilted either clockwise or anti-clockwise.Observers had limited time to compare the tilt ofthe foveal and peripheral targets. The tilt and con-trast of all four patterns was independently per-turbed with zero-mean Gaussian noise, refreshedevery two video frames (∼24 ms). The tilt offsetwas sufficiently small that this was a foveal task.That is, in order to perform the task observers hadto (i) analyse the tilt of the foveal target; (ii) sac-cade to the peripheral target from the non-targetson the basis of luminance contrast; (iii) fixate theperipheral target and analyse its orientation; and(iv) respond whether the foveal and peripheral tar-


Test sequence (2 frames/image)

...

Preview (random foreperiod)

Fixation

Target

Q1: Orientation at initial fixation? R1: Right

Q2: Target same or different orientation?R2: Different

Figure 2. Illustration of the paradigm. Trials startwith a preview of variable duration. The previewis replaced by a sequence of test images, with eachimage shown for 2 video frames (∼24 ms/image),for a total duration of ∼750 ms, or 32 images at an85 Hz monitor refresh rate. The peripheral targetis signalled by its higher average luminance con-trast (straight up in this figure). The mean targetand non-target contrasts are equidistant from thepreview contrast. The fixation pattern remains atthe preview contrast. The contrast of all patternsis perturbed independently with zero-mean Gaus-sian noise (σ = 0.15). The fixation pattern canbe tilted clockwise or anti-clockwise from vertical(clockwise in this example). The target pattern canbe tilted in the same or a different direction (anti-clockwise in this example). The direction of tilt ofthe remaining patterns is determined randomly andindependently, so that their tilt conveys no infor-mation about the likely target orientation (i.e. theorientations of the peripheral patterns are uncorre-lated). The orientation of all four patterns is alsoperturbed independently with zero-mean Gaussiannoise (σ = 6◦). The mean pattern tilt was either 1◦

or 2◦ (as in this figure).

gets were tilted in the same or different direction.In addition, they also reported the tilt direction ofthe foveal target.

While this was a challenging task, the overall‘same-different’ judgement was performed abovechance by 7/8 observers (based on the 95% bino-mial confidence intervals). Averaged across ob-servers the overall accuracy was 60%. Accuracyof foveal target tilt discrimination was well abovechance for all observers, with a mean of 77%correct. The peripheral target was correctly fix-ated with the first saccade on 63% of the trials(note that chance performance here correspondsto 33%; all observers were above chance on thismeasure). Figure S1 of the Supporting Informa-tion summarises a number of performance mea-sures along with the individual observer data.

Temporal integration windows for foveal analy-sis and peripheral selection

To identify the foveal integration window, wefocus on the tilt discrimination of the foveal tar-get. To identify the peripheral integration windowfor saccade target selection, we focus on which ofthe three peripheral patterns was selected for thefirst saccade. The logic of the noise classificationapproach is straightforward: If a noise sample ata particular point in time was processed and usedto drive behaviour, it should be predictive of be-haviour. By assessing to what extent the noise atvarious points in time is predictive of behaviour,we obtain an estimate of the underlying temporalweighting function that observers use to performa particular task [27]. We performed this analy-sis aligned on the onset of the test sequence (‘dis-play aligned’) and aligned on the onset of the firstmovement (‘saccade aligned’).

We start with foveal analysis. Suppose that, forexample, the true direction of tilt of the foveal tar-get on a given trial is +1◦ (clockwise). Supposethe observer is particularly sensitive to the first twosamples presented after the onset of the test se-quence (corresponding to the first ∼50 ms of thesequence). Due to random sampling, the orienta-tions presented during this interval are −3◦ and 0◦.The mean orientation over this interval is negativeand the observer signals that the tilt of the foveal


target was anti-clockwise. This response would beclassed as an error in our analysis. As a first stepthen, we averaged the orientation noise traces ofthe foveal target separately for correct and errordecisions to infer the temporal interval used by theobserver to make a decision. Before averaging, wesubtracted the true mean orientation from the se-quence of tilt values, so that we were left with thenoise samples only and 0 corresponds to the “true”mean tilt. Noise samples that tilted the pattern fur-ther in its nominal (mean) orientation were given apositive sign (e.g. more clockwise for a clockwisepattern); samples that tilted the pattern in the op-posite direction from the nominal orientation weregiven a negative sign. Trials with different levelsof mean tilt (1◦ or 2◦) were pooled together in thisanalysis.

The top row of Figure 3 shows these “classi-fication images” for a single observer. The left-hand panel shows the display aligned traces for thewhole trial duration; the right-hand panel showsthe traces aligned to saccade onset. Where the twocurves differ we have evidence that the noise in thestimulus influenced the decision: During this inter-val, noise samples that tilted the pattern away fromits true direction were more likely to induce an er-ror and noise samples that tilted the pattern evenfurther in the true direction were more likely to in-duce a correct decision. The foveal nature of thisdiscrimination is readily apparent in these traces,particularly when aligned on eye movement on-set: Before movement onset the orientation noiseclearly influenced the perceptual tilt judgement,but after the movement the noise had a much lesspronounced effect on the decisions. Aligned ondisplay onset, the traces converge rather graduallydue to the variability in the duration of the initialfixation.

Next consider peripheral selection. The analy-sis here is more complex, because there were threepatterns to choose from. However, the logic isvery similar. Consider an observer who utilises theinformation presented between 100-150 ms afteronset of the test sequence. If during this interval

0 150 300 450 600 750−1.5

−1

−0.5

0

0.5

1

1.5

time wrt display onset (ms)

orie

ntat

ion

nois

e (d

egs;

+ve

: in

mea

n di

rect

ion) display aligned

correcterror

−400 −200 0 200 400 −1.5

−1

−0.5

0

0.5

1

1.5

time wrt saccade onset (ms)

saccade aligned

0 150 300 450 600 750−0.02

−0.01

0

0.01

0.02

time wrt display onset (ms)

cont

rast

noi

se

−400 −200 0 200 400 −0.02

−0.01

0

0.01

0.02


chosen non−targetignored targetignored non−target

Figure 3. Raw temporal classification images forfoveal identification (orientation noise, top row)and peripheral selection (bottom row) for one ob-server. Note that display onset refers to the start ofthe noisy test image sequence; it is not the same asstimulus onset, due to the preview during which allpatterns were already present (see Figure 2). Thethree triangles in the display aligned panels cor-respond to the 25th, 50th (median), and 75th per-centiles of the fixation duration distribution. Thegrey shaded box in the saccade aligned plots cor-responds to the mean saccade duration for this ob-server.

one of the non-targets happens to have a particu-larly high luminance contrast, the observer may bemore likely to select that pattern for the next fixa-tion [7]. In other words, the sequence of contrastvalues in the periphery should be predictive of theobserved fixation behaviour. For this analysis, weonly considered erroneous saccades directed to anon-target. Figure S2 in the Supporting Informa-tion shows that little insight is gained from correctsaccade trials in the identification of the integrationwindow for peripheral selection.


The bottom row of Figure 3 shows the aver-age noise traces from all three peripheral locationspreceding these saccades, for the same observer.Of critical interest is the comparison between theignored target and the chosen non-target. Asexpected, errors in peripheral selection occurredwhen the non-target happened to be relatively highin contrast and/or the target was relatively low incontrast. Again, where the two curves differ wehave evidence that noise in the stimulus influencedthe decisions. Of course in this instance, noise thatoccurred after saccade onset cannot – by definition– influence saccade target selection.

To compare the temporal processing windowsunderlying foveal analysis and peripheral selectionmore directly, we calculated the area under the re-ceiver operating characteristic (ROC) curve at eachpoint in time, given two distributions of noise sam-ples: correct versus error for foveal analysis; cho-sen non-target versus ignored target for peripheralselection (the thick black and grey lines in Figure3). This measure quantifies the separation betweentwo distributions as the probability with which apair of noise samples can be accurately assigned tothe two stimulus or response classes by compari-son to a criterion value [28]. When two distribu-tions lie completely on top of each other, this clas-sification cannot be made above chance level. Asthe distributions separate, classification becomesmore accurate. This measure allows for directcomparison between foveal analysis and periph-eral selection on a meaningful scale (probability),that incorporates the variability of the noise sam-ples within each stimulus or response class (notincluded in Figure 3).

The area under the ROC curve may be computednon-parametrically, so that we do not have to makeany assumptions about the distributions of noisevalues [29]. While the distributions that generatedthe external noise were Gaussian, there is no guar-antee that they will still be Gaussian once condi-tionalised on stimulus or response class. For exam-ple, extreme noise values drawn from the tails ofthe distribution are more likely to generate errors

in tilt discrimination than noise values closer to themean. Our analysis here is similar to that used insingle-cell neurophysiology to quantify the extentto which single neurons can distinguish betweentwo stimuli (where distributions of firing rates arefrequently non-normal) [30, 31, 32].

The ‘Materials & Methods – ROC analyses’ sec-tion explains in detail how the analysis was per-formed. In brief, we iterated a criterion from asmall value (where all noise values from both dis-tributions lie to the right of the criterion) to a largevalue (where all the noise values lie to the left ofthe criterion). For each criterion value, we com-puted the proportion of noise values from the er-ror (foveal analysis) and ignored target (peripheralselection) distributions that were greater than thecriterion value. In addition, we computed the pro-portion of noise values from the correct (fovealanalysis) and chosen non-target (peripheral selec-tion) distributions that were greater than the cri-terion. Plotting one proportion (correct or chosennon-target) against the other (error or ignored tar-get) traces out the ROC curve. This curve was thennumerically integrated to find the “classificationaccuracy”.

Figure 4 shows this accuracy measure as a func-tion of time, averaged across all 8 observers. Thetemporal integration window now corresponds tothe region where the classification accuracy isgreater than chance. The uptake of information forfoveal analysis and peripheral selection occurredlargely in parallel. In particular, foveal orientationand peripheral contrast signals were monitoredright from the onset of the test sequence: Classi-fication accuracy for both foveal analysis and pe-ripheral selection is clearly above chance at the be-ginning of the test sequence. Aligned on move-ment onset, foveal information was processed rightup to the onset of the first saccade and suppressedduring the saccade (indicated by the grey shadedbox). The uptake of peripheral information ceasedsome 60-80 ms before the saccade, which is com-patible with other estimates of a so-called saccadicdead time [6, 33, 7, 8]. This dead time corresponds


to the period before movement onset during whichnew visual information can no longer modify theupcoming movement. It is the “point of no return”in saccade programming. Our data clearly showthat foveal processing continued during this deadtime. A final feature is that the function for fovealanalysis recovers from saccadic suppression andis raised above the baseline almost straight awayafter movement offset. In other words, observerscontinued to process some orientation informationfrom the previously fixated location, even thoughthis location was now in the periphery.

0 150 300 450 600 750

0.5

0.55

0.6

0.65

time (ms) wrt display onset

clas

sific

atio

n ac

cura

cy (a

rea

unde

r RO

C)

display aligned

−400 −200 0 200 400

0.5

0.55

0.6

0.65

time (ms) wrt saccade onset

saccade alignedfoveal analysisperipheral selection

Figure 4. Noise classification accuracy for fovealidentification and peripheral selection, averagedacross 8 observers. In the saccade aligned panel,the average movement duration is shown by thevertical shaded box. The shaded region aroundthe functions corresponds to the 95% confidenceinterval across the subject pool. Note that in thesaccade aligned plot, fewer trials contribute to theextreme time points (i.e. long before movementonset and long after movement offset). To align thenoise samples on movement onset, we assigned thesample during which the movement started to time0. This relatively crude alignment means that the“true” onset of the saccade relative to the start ofthe noise sample is accurate within the duration ofan individual noise frame (i.e. ∼24 ms). Given thelarge amount of data collected for each observer,the average starting point of the movement will lienear the mid-point of the noise frame.

On the whole, classification accuracy is lower

for peripheral selection than for foveal analysis.This finding implies that the peripheral contrastnoise was less predictive of the upcoming sac-cadic decision, compared to the predictive valueof the foveal orientation noise for tilt discrimina-tion. Observers may simply be less sensitive tothe peripheral contrast information [34]. Reducedcontrast sensitivity in the periphery will diminishthe influence of the external noise on behaviourand thereby reduce its predictive value. Alterna-tively, it is possible that different amounts of in-ternal noise are added by the sensory apparatus tothe foveal orientation and peripheral contrast sig-nals [35, 36]. Internal noise dilutes the influenceof external noise and thereby reduces its predictivevalue. We cannot say whether any such differencesin internal noise depend on the location in the vi-sual field (foveal vs. periphery) or on the specificfeature dimensions involved (contrast vs. orienta-tion).

Interaction between foveal analysis and periph-eral selection

Having demonstrated the temporal uptake of in-formation for foveal analysis and peripheral selec-tion, we are now in a position to address the inter-action between the two tasks. Foveal tilt discrim-ination difficulty was varied at two levels, deter-mined by the mean offset from vertical. We referto these conditions as high load (1◦) and low load(2◦). As illustrated in Figure 1 (panels C and D),we may expect this variation in load to influencethe uptake of information from the fovea and theperiphery. Such changes in the uptake of infor-mation may be identified using our noise classi-fication approach. In particular, we might expectchanges in the width and/or amplitude of the tem-poral integration windows for foveal analysis andperipheral selection, with variations in foveal load.

First we consider the behavioural results underthe two levels of foveal processing difficulty. Ta-ble 1 summarises the relevant behavioural perfor-mance measures. When foveal load was low, ac-curacy improved by ∼20%. There was a very


small effect on the duration of the first fixation,with observers moving away from the foveal tar-get slightly earlier when the processing load waslow (statistically the effect is negligible [37]). Theaccuracy of peripheral selection was completelyunaffected by the difficulty of foveal processing.In summary, making foveal processing easier didclearly benefit tilt identification, but did not affecteye movement behaviour: both fixation durationand target selection accuracy were essentially con-stant with the variation in foveal load.

Next, we examined the underlying temporal in-tegration windows, by performing the noise clas-sification analysis separately for the two levels ofprocessing load. Figure 5 shows the classifica-tion accuracy for each level of foveal load. Thesecurves largely overlap (and fall in each other’s con-fidence interval) and there is no evidence for sys-tematic and reliable differences. These data sug-gest that observers did not adjust their uptake of in-formation in response to variations in the quality offoveal evidence. The lighter foveal processing loaddid not affect the amplitude, shape, or width of thetemporal processing windows for foveal analysisand peripheral selection. Supplementary Figure S3shows that this result held regardless of whetherthe foveal load was varied randomly between trialsor systematically between blocks.

These results suggest a degree of independencebetween foveal analysis and peripheral selection.To test independence between these processesmore thoroughly, we performed (part of) a classicdual-task analysis. That is, we examined to whatextent performance on one task (foveal analysis inthis case) suffered from the addition of another (pe-ripheral selection). The logic goes as follows. Ifboth tasks share a bottleneck or limited capacityresource, having to perform the peripheral selec-tion task concurrently with foveal analysis may beexpected to impair the latter. If the two tasks pro-ceed independently, foveal analysis of tilt would bejust as accurate with or without the peripheral se-lection demand. In that case, performance on thefoveal analysis task in isolation should allow us to

0 150 300 450 600 750

0.5

0.55

0.6

0.65

clas

sific

atio

n ac

cura

cy (a

rea

unde

r RO

C)

display aligned

0 150 300 450 600 750

0.5

0.55

0.6

0.65


−400 −200 0 200 400

0.5

0.55

0.6

0.65

saccade aligned

−400 −200 0 200 400

0.5

0.55

0.6

0.65


foveal analysis

peripheral selection

high load (1°)low load (2°)

Figure 5. Noise classification accuracy for fovealidentification (top row) and peripheral selection(bottom row), averaged across 8 observers. Eachpanel contains a separate function for the twolevels of foveal processing load. In the saccadealigned plots (right column), the saccade durationis shown by the shaded vertical box. The shadedregions around the functions show the 95% confi-dence intervals.

predict perceptual discrimination performance inthe main, comparative task.

One issue to consider in this regard is time. Wehave shown that the uptake of information fromthe fovea occurred throughout the entire fixationduration. Given the dynamic nature of the ex-ternal noise, the accuracy of foveal tilt discrim-ination will depend on the duration of the fixa-tion/integration epoch. For any one individual ob-server, we therefore need to be able to predict theaccuracy of foveal discrimination at a time scalethat is relevant to that observer. For this reasonwe measured foveal orientation discrimination ac-curacy as a function of time in a separate experi-ment (see ‘Materials & Methods – Single patterntilt judgements’).

Figure 6A shows accuracy for the two levels of


Table 1The effect of foveal processing load on behaviour (averaged across 8 observers)performance measure high load:1◦ low load:2◦ CI-1 CI+ BF10

2

foveal target tilt 0.70 0.85 -0.16 -0.14 2.24 × 105

first fixation duration 421 414 -1 14 1.49saccade target accuracy 0.63 0.62 −3 × 10−3 0.02 1.07195% Confidence intervals on the difference between low and high load. 2Bayes Factor in favour of the‘alternative’ hypothesis that there is a difference between the two load conditions.

foveal processing difficulty, averaged across ob-servers. Performance clearly improves with pre-sentation time, up to a plateau. Overall accuracywas higher for the lower foveal load and the twocurves appear to be separated by approximately aconstant. We constructed these functions for eachindividual observer. We then plugged the observedmean first fixation duration from the main exper-iment into the function corresponding to a partic-ular observer and foveal load. We simply foundthe predicted probability correct for a given fixa-tion duration through linear interpolation betweentwo data points. Figure 6B shows the correspon-dence between predicted tilt accuracy based on sin-gle task performance and tilt discrimination in themain task with a concurrent peripheral selectiondemand.

The addition of a peripheral selection demand,if relying on a common bottleneck or limited ca-pacity resource, should have lowered tilt discrimi-nation in the main task. As a result, we would haveexpected the data points to lie below the diagonal.Clearly, that is not the pattern we found. Instead,the data points scatter around the identity line, withno obvious systematic offset. Averaged across thetwo levels of foveal difficulty (to ensure indepen-dence between observation pairs), the correlationis r(8) = .7, p = .03 (one-tailed). A Bayesian t-teston the difference between the averaged observedand predicted scores [38], resulted in a Bayes Fac-tor of 0.54 (effectively no evidence against the null[37]). Taken together then, the behavioural perfor-mance data, temporal integration windows and thedual-task analysis all strongly support the conclu-sion that peripheral selection was performed at no

0 200 400 600 8000.5

0.6

0.7

0.8

0.9

1

presentation time (ms)

prop

ortio

n co

rrec

t


0.5 0.6 0.7 0.8 0.9 1

0.5

0.6

0.7

0.8

0.9

1

obse

rved

acc

urac

y

predicted accuracy

A B

Figure 6. Comparison of foveal tilt discrimina-tion under single and dual-task conditions. A –Single-task performance. Observers viewed a sin-gle pattern at fixation, which fluctuated in orien-tation and contrast in exactly the same way as thefoveal target in the main experiment. After a vari-able interval, the pattern was extinguished and theobserver generated a vertical, upward saccade to aGaussian noise-patch. Accuracy is averaged acrossobservers; the error bars are within-subject stan-dard errors of the mean. B – Dual-task tilt perfor-mance (i.e. with a concurrent peripheral selectiondemand) as a function of single-task performance.Single-task accuracy was found by plugging themean first fixation duration into the functions re-lating tilt discrimination accuracy to viewing time(as in panel A).

cost to the analysis of the currently fixated item.

Discussion

The visual environment is explored by discretelysampling it through active gaze shifts. During a pe-riod of stable fixation, the information at the cur-rent point of gaze is analysed in fine detail using


foveal vision. At the same time, decisions aboutwhere to fixate next need to be made. We havedeveloped a dual-task noise classification method-ology which enabled us to identify what informa-tion was processed, from which spatial location,and at what point in time during an individual fixa-tion. Using this methodology, we found that fovealanalysis and peripheral selection largely proceed inparallel and independently from each other. Fovealanalysis spans essentially the entire fixation du-ration and continues all the way up to movementonset. The extraction of information from the pe-riphery for saccade target selection stops sometimeearlier. During this dead time, information contin-ues to be processed from the fovea. Variation inthe difficulty of foveal processing did not affectthe uptake of information from the fovea or theperiphery. In the following sections, we examinethese claims about parallelism and independencein more detail.

Foveal analysis and peripheral selection occurin parallel

One interpretation of our results is that at leastfor some period of time, attention is split betweenmultiple locations: attention is needed to extractorientation information from the central pattern,and contrast information from the peripheral pat-terns [10]. The noise classification analysis sug-gests the extraction of this information proceedsin parallel. However, it could be argued that ourmethodology is not able to distinguish betweenparallel processing and rapid random serial shift-ing of attention [39, 40, 9]. The temporal integra-tion windows are estimated over many trials. Onany one given trial, attention could rapidly shift be-tween the four patterns in the display. Given somevariability in the order and speed with which indi-vidual items are attended, it is likely that each lo-cation will have been sampled at a particular pointin time in some trials. Taken across many trials, itwould then appear that information was extractedfrom the fovea and periphery at the same point intime.

The strongest evidence against rapid serial shift-ing comes from our dual-task analysis. In the sin-gle pattern tilt judgements there were no peripheralpatterns to shift attention to. As such, attentionwould, presumably, have remained on the fovealtarget at all times. In the dual-task condition,however, attention would have moved between thefoveal target and all three peripheral items. Forthe same fixation duration, the proportion of timefor which the foveal target was attended should bedrastically reduced in the dual-task condition. Assuch, tilt discrimination should have been superiorin the single-task condition. On the contrary, theanalysis shown in Figure 6 shows that foveal targettilt discrimination was essentially the same in bothsingle and dual-task conditions. Based on theseconsiderations, the most parsimonious explanationof the temporal integration windows identified inthe present study is that foveal analysis and pe-ripheral selection started together and proceededin parallel.

However, evidence against a rapidly shifting at-tentional focus does not necessarily provide evi-dence that attention was divided. Attention is typi-cally thought of as a unitary mechanism: all visualfeatures that fall within the spotlight are enhanced.Indeed, one of the proposed primary functions ofattention is to bind different features to the sameobject in the focus of attention [40]. Some aspectsof our data (Supporting Information – section 4and Figure S4) argue directly against such a uni-tary mechanism. Limited processing of tilt at thefuture fixation position occurs once peripheral in-formation about contrast has been processed andthe saccade target has been (or is being) selected.If attention had already “visited” peripheral loca-tions for the purpose of deciding which pattern hadthe higher contrast, why would information aboutorientation processed along the way not contributeto subsequent judgements of peripheral target tilt?

Indeed, we see no need to invoke an attentionalspotlight to account for the highly selective uptakeof foveal and peripheral information. In order toperform the task, observers may adaptively choose


which upstream sensory channels to monitor. Forinstance, for foveal tilt discrimination observersmight monitor central visual mechanisms with off-vertical orientation preferences. Tilt judgementsmay involve accumulation of a decision variablethat tracks the difference in the neural response ofneurons with clockwise and anti-clockwise orien-tation preferences [41, 42]. The sign of this in-tegrated difference variable may then be used tomake the tilt judgement [43, 44, 45]. For pe-ripheral selection, observers could monitor periph-eral mechanisms tuned to vertical orientations thatscale their response with pattern contrast. The eyemovement decision may be based on integratingand comparing mechanisms that represent differ-ent locations in the visual field, with the saccadetarget being the location that triggered the maxi-mum integrated response [46, 47].

Foveal analysis and peripheral selection occurindependently

The absence of a dual-task cost in conditionswith variable foveal load is striking, given previ-ous demonstrations of tunnel vision and alteredperipheral processing with changes in foveal load[12, 13, 14, 16]. These previous findings are con-sistent with a reduction in peripheral gain underconditions of high foveal load. Why did we see noevidence at all for such a gain change?

One possible reason is that the task-relevant vi-sual features were rather basic and – perhaps moreimportantly – different for foveal analysis and pe-ripheral selection. Concurrent, un-speeded visualdiscrimination tasks in the fovea and periphery areperformed without interference when the two tasksinvolve different feature dimensions (e.g. colourand luminance [48]), or when the peripheral taskinvolves discrimination between well-learned, bi-ologically relevant categories (e.g. animal versusnon-animal discrimination [49]). Dual-task costsare observed when the discrimination in the foveaand periphery involves the same dimension or lesswell-practiced feature combinations [e.g. ‘T’ ver-sus ‘+’ discrimination [49]). It is possible then that

peripheral information can be analysed at no costto foveal processing when sufficiently specialiseddetectors are involved so that no or little bindingacross feature channels is needed.

The limitations of peripheral visual processing,in conjunction with the clutter of natural visualscenes, are such that eye guidance by complexcombinations of features may not always be pos-sible [50, 51, 52] (but see [53]). Indeed, the pri-mary reason to fixate a region in the visual field isto extract more complex and detailed informationfrom that location with the high resolution fovea.For example, in reading, low spatial frequency in-formation about word boundaries is used to selectthe target for the next saccade [54]. However, lowspatial frequency information about the coarse out-line of words is effectively useless when it comesto identifying those words, which relies on highspatial frequency shape information and combin-ing elementary features [55]. The analysis of thisinformation requires the fovea.

Another possible reason for the independenceobserved in the present study is that foveal analysisand peripheral selection must be one of the mostextensively practised dual-tasks humans (and otherfoveated animals) are confronted with. We shiftgaze about three times every second during ourwaking hours [56], although not all of these shiftsare visually guided. Maintaining vigilance andawareness of the peripheral visual field is clearlyimportant for our survival (e.g. navigation and lo-comotion [57]; detecting predators or other kindsof potential hazards [58]). Combining foveal ob-ject identification with some basic peripheral fea-ture processing to enable rapid orienting responsesmay just be a particular dual-task that the brain hasadapted – over the course of evolution or within anindividual’s lifetime – to perform without interfer-ence [59].

Conclusions

Much of the neural and behavioural work onattention and eye movement control has little tosay about the foveal component of active gaze be-


haviour. Most studies and models are concernedwith the system’s response to peripheral visual in-formation, typically with minimal foveal process-ing demands. The majority of models of visualsearch and scene perception focus exclusively onthe selection of fixation locations [60, 61, 62, 63](see [50] for a review). Some models of search arebased on template matching and assume that a tar-get template is applied across the visual field inde-pendently and in parallel (e.g. [61, 63]), but to ourknowledge this assumption has not yet been testedexperimentally. In terms of the underlying neuro-physiology, many studies have charted the compe-tition between neurons representing potential pe-ripheral target locations, again in the absence ofany foveal processing demand (for a review, see[64]). The lack of consideration to the foveal pro-cessing demands is striking, given that the primaryreason to shift gaze to a certain location is to ex-tract the information at that location with greaterresolution. Any model of eye movement controlneeds to solve the same problem, regardless of thedomain of application: how are foveal analysisand peripheral selection coordinated? Our studyprovides a default starting position on the issue.Foveal analysis and peripheral selection occur inparallel and independently.

Materials and Methods

Observers

One group of four observers experienced the twodifferent levels of foveal processing load randomlyinter-mixed. Another group of four observers ex-perienced the two load conditions in a blockedmanner. It turned out that this variation in pre-sentation mode did not affect behaviour at all (seealso Supplementary Figure S3), which is why wehave presented the data from all eight observers to-gether. Five observers were female; the age rangeacross all observers was 21-33. All subjects hadnormal or corrected vision. They were paid fortheir help at a rate of £7.5//hr. The study was ap-proved by the local Faculty Ethics Committee and

complied with the principles of the Declaration ofHelsinki (2008 revision). Each observer was givenone initial session on the comparative task as prac-tice. The study then started with one initial sessionin which time versus accuracy curves were mea-sured for the two levels of foveal load (without aperipheral selection demand), followed by 15 ses-sions of the comparative task, and one final timeversus accuracy measurement. In total then, eachsubject completed 18 ∼1 hr sessions on differentdays. In the comparative tilt judgement task, ob-servers performed 4 blocks of 96 trials in a session,for a total of 5760 trials.

Stimuli and equipment

Stimuli were generated using custom writtensoftware in Matlab (The Mathworks, Inc.) usingPsychToolbox 3.08 [65, 66, 67]. They were pre-sented on a Viewsonic G225f 21” CRT monitor,running at 85 Hz with a spatial resolution of 1024x 768. To create fine steps in luminance contrast,the graphic card output was enhanced to 14 bitsusing a bits++ digital video processor (CambridgeResearch Systems, Ltd.). The range between theminimum and maximum luminance (determinedby the maximum contrast pattern, i.e. the targetcontrast + twice the standard deviation of the con-trast noise distribution) was sampled in 255 stepsusing a linearised look-up table. One additionalgrey level was used for the fixation point and cali-bration targets. The screen was set to mid-grey (47cd m−2).

Eye movements were recorded at 1000 Hz usingthe EyeLink 2k system (SR Research Ltd.). Sac-cades were analysed offline using velocity and ac-celeration criteria of 30◦s−1 and 8000◦s−2. The eyetracker was calibrated using a grid of 9 points at thestart of each block of trials. The calibration targetwas a ‘+’ with each leg measuring 0.6◦×0.1◦. Eachtrial started with presentation of a central fixationpoint (identical to the calibration target). The stim-uli were presented automatically as soon as the ob-server’s fixation remained within 1.5◦ of the centrefor 500 ms.


Comparative tilt judgement task. Displaysconsisted of four Gabor patterns, with a spatial fre-quency of 2 cycles deg−1. The circular standarddeviation of the Gaussian envelope was 0.5◦. Thepatterns were in sine phase. One pattern replacedthe initial fixation point in the centre of the screen.The three remaining patterns fell on an imaginarycircle around fixation, at an eccentricity of 8◦. Thepatterns were always at 90◦ angles, but could ap-pear in one of four configurations: top (as in Fig-ure 2), left, bottom and right. The variation in theconfiguration is included to discourage observersfrom developing stimulus-independent saccade bi-ases (e.g. always saccade straight up). All four pat-tern configurations appeared equally often within ablock of 96 trials.

During the preview, all patterns were stationaryin both contrast and (vertical) orientation. TheMichelson contrast of the underlying sinusoid,Lmax−LminLmax+Lmin

, was set to 0.4. The preview duration wasapproximately distributed according to a shiftedand truncated exponential (minimum: 235 ms;mean: 490 ms; max: 1 s). The preview was fol-lowed by a ∼750 ms test period. The mean con-trast of the foveal target remained at 0.4. The meanperipheral target contrast was 0.475; that of theperipheral non-targets was 0.325. These contrastvalues were perturbed with Gaussian noise with astandard deviation of 0.15. On half the trials thetilt offset of the foveal target was clockwise; onthe other half the offset was anti-clockwise. Onhalf the trials the tilt offset of the peripheral targetwas the same and on the other half the offset wasdifferent to that of the foveal target. The offsetsof the two peripheral non-target patterns were cho-sen randomly and independently from trial-to-trial.The orientation of all four patterns was perturbedwith Gaussian noise with a standard deviation of6◦. The noise perturbations in both contrast andorientation were applied independently to all fourpatterns every 2 video frames. A single noise sam-ple or frame lasted ∼24 ms.

Observers were simply told about the perceptualjudgements they had to make. We did not give

any specific eye movement instructions. The con-straints of the task were such that observers hadto move their eyes to the peripheral target in or-der to identify its orientation and compare it tothe initially viewed foveal target. The duration ofthe test period was sufficiently short that observershad to be selective in where to direct their first eyemovement: there simply was not enough time toinspect each pattern in the display with foveal vi-sion and estimate its tilt direction accurately (seeFigure 6A).

Single pattern tilt judgements. Observersperformed two separate sessions in which theyjudged the tilt of a single pattern. In this task, theywere presented with a vertically oriented, station-ary Gabor at fixation with a luminance contrast of0.4. The peripheral patterns were not shown. Af-ter the random foreperiod, the fixation point wasremoved and the pattern was tilted away from ver-tical by 1◦ or 2◦ (randomly inter-mixed or blockedin the two different subject groups). As in the mainexperiment, the patterns were perturbed with tem-poral orientation and contrast noise. After a vari-able delay (4, 8, 16, 32, 64 frames, correspondingto ∼47, 94, 188, 376 and 753 ms), a peripheral pat-tern appeared straight up from the fixated pattern(i.e. in the 90◦ location in Figure 2). This periph-eral pattern was a Gaussian windowed, high con-trast patch of noise, created by adding zero-meanGaussian luminance noise to the background lu-minance (as a proportion of the maximum screenluminance, the mean luminance of the noise patchwas 0.5 and the standard deviation was 0.16). Thecircular standard deviation of the window functionwas the same as that used for the Gabor patches.At the same time, the fixated pattern disappeared,which signalled to the observer to make a verticalupward saccade to fixate the noise patch.

The requirement to make a saccade was includedto keep the motor demands as close as possible tothe main comparative tilt judgement task, withoutinvoking a peripheral selection demand. The noisepatch also served as a post-saccadic foveal maskthat was broadband in orientation and spatial fre-


quency. Upon successful completion of the sac-cade, the observer indicated the orientation of thefixated pattern with an un-speeded manual buttonpress. Note that the saccade target always appearedin the same location, and that the “go” signal wasthe offset of the fixation stimulus. As such, thistask required no or minimal peripheral processing.In each session, observers performed five blocks of100 trials (10 repetitions of each presentation timefor the two levels of foveal processing load).

Data analysis

Eye movement data. We were predominantlyinterested in the first saccade that brought the eyesfrom the foveal target to one of the three periph-eral items. We only included trials in which thatsaccade was generated after the onset of the testsequence and started within 2◦ of the foveal tar-get. Provided the saccade had a minimum ampli-tude of 4◦ it was assigned to the nearest periph-eral pattern. After filtering the data set in this way,the total number of included trials ranged between5299 and 5760 across observers.

Classification images. Foveal analysis. Trialswere separated by accuracy of the tilt judgementof the foveal target. For each trial, we subtractedthe “true” mean orientation from each vector of Mtilt samples (M = 32). The resulting noise valueswere given a sign so that positive values indicatedsamples tilted in the direction of the nominal (true)tilt, and negative values represented samples tiltedaway from the nominal direction. We then simplyaveraged the noise samples across the trials in eachresponse class. Trials from the two foveal loadconditions were pooled together in this analysis. Astandard classification image would correspond tothe difference between the correct and error tracesin Figure 3 [26]. Instead, we performed an ROCanalysis over time so that the difference takes thevariability around the average noise traces into ac-count and yields a result in meaningful units thatcan be directly compared across tasks (see below).

To align the data on movement onset, we foundthe noise sample during which the movement was

initiated. To find this sample, we simply divide thefixation duration (millisecond resolution) by theduration of a single noise frame and round the re-sult up to the nearest integer sample. The vector ofM noise values is then “copied” into a larger vectorof 2M − 1, with time 0 (movement onset) assignedto element m = M in this expanded representation.Empty cells (long before movement onset and longafter movement onset) were set to ‘Not-a-Number’in Matlab and were not included in the calculationof the average noise traces and ROCs.

Peripheral selection. Only trials with an inac-curate saccades directed to a non-target were in-cluded in this analysis. For each of these trials wehave three vectors of M contrast samples, corre-sponding to the ignored peripheral target, the cho-sen non-target and the ignored non-target. We sub-tracted the relevant mean contrast from each ofthese three traces, before averaging them. Thealignment on movement onset occurred in exactlythe same way as in the analysis of the orientationnoise.

ROC analyses. The first steps of this analy-sis are identical to those in the classification imageanalyses described above, prior to the final aver-aging step. For the uptake of peripheral contrastinformation we did not include the ‘ignored non-target’ noise values. As illustrated in Figure 3,these traces hovered around 0 and did not appear tocontribute to selection of the next fixation point. Aconvenient way to represent the data at this stage isas two matrices. For instance, for display alignedfoveal analysis, we have one N1 ×M matrix for er-ror trials and one N2 × M matrix for correct trials.We refer to these matrices as X1 and X2 respec-tively. In the case of peripheral selection, the ma-trices would contain the contrast noise values forthe ignored target and chosen non-target respec-tively. Note that the typical values for N1 and N2

were well into the hundreds (and frequently wellin the thousands).

Consider each time sample j in turn, wherej = 1, . . . ,M in the display aligned analyses andj = 1, . . . , 2M − 1 in the saccade aligned analyses.


There are two vectors of noise values: g1 = X. j1

and g2 = X. j2 , where the dots in the superscript in-

dicate that we take the values from all rows in thematrix. Only the real valued samples from X aretaken in the construction of g. A non-parametricROC curve is created by evaluating the proportionof noise samples that are greater than a criterionvalue [28, 29, 68]. The criterion value is changedfrom some small value (where both distributionsfall to the right of the criterion) to some relativelylarge value (where both distributions fall to the leftof the criterion). The extreme values of the crite-rion fix the start and endpoints of the ROC curve at(1, 1) and (0, 0).

In between these extreme values, the curve isevaluated at a further 20 criterion values, linearlyspaced between the minimum and maximum val-ues across the two distributions. For each crite-rion value, we evaluated the proportion of noisevalues in vectors g1 and g2 that were greater thanthe criterion. The corresponding point on the ROCcurve is (p1k, p2k), for k = 1, . . . , 22. The area un-der the ROC curve was then computed by simplenumerical integration. While this non-parametric,numerical procedure is relatively brute-force, wehave verified that these area estimates were stableand no longer dependent on the number of crite-rion values chosen. Note that in standard signal-detection theoretic terms, we treat distribution g2

(correct orientation noise samples, chosen non-target contrast samples) as the “signal” and distri-bution g1 (error orientation noise samples, ignoredtarget samples) as “noise”.

References

[1] Curcio CA, Sloan KR, Kalina RE, Hendrick-son AE (1990) Human photoreceptor topog-raphy. Journal of Comparative Neurology292(4):497–523.

[2] Daniel P, Whitteridge D (1961) The repre-sentation of the visual field on the cerebralcortex in monkeys. The Journal of Physiol-ogy 159(2):203–221.

[3] Rovamo J, Virsu V (1979) An estimationand application of the human cortical magni-fication factor. Experimental Brain Research37(3):495–510.

[4] Ballard DH, Hayhoe MM, Pook PK, RaoRPN (1997) Deictic codes for the embodi-ment of cognition. Behavioral and Brain Sci-ences 20(4):723–767.

[5] Botvinick M, Plaut DC (2004) Doing with-out schema hierarchies: A recurrent connec-tionist approach to normal and impaired rou-tine sequential action. Psychological Review111(2):395.

[6] Becker W, Jürgens R (1979) An analysis ofthe saccadic system by means of double stepstimuli. Vision Research 19(9):967 – 983.

[7] Caspi A, Beutter BR, Eckstein MP (2004)The time course of visual information accrualguiding eye movement decisions. Proceed-ings of the National Academy of Sciences ofthe United States of America 101(35):13086–13090.

[8] Ludwig CJH, Mildinhall JW, Gilchrist ID(2007) A population coding account for sys-tematic variation in saccadic dead time. Jour-nal of Neurophysiology 97(1):795–805.

[9] Eriksen C, Yeh Y (1985) Allocation of at-tention in the visual field. Journal of Exper-imental Psychology: Human Perception andPerformance 11(5):583–597.


[10] Müller M, Malinowski P, Gruber T, HillyardS (2003) Sustained division of the attentionalspotlight. Nature 424(6946):309–312.

[11] Jans B, Peters J, De Weerd P (2010) Visualspatial attention to multiple locations at once:The jury is still out. Psychological Review117(2):637–682.

[12] Mackworth NH (1965) Visual noise causestunnel vision. Psychonomic Science 3(2):67–68.

[13] Ikeda M, Takeuchi T (1975) Influence offoveal load on the functional visual field. Per-ception & Psychophysics 18(4):255–260.

[14] Henderson JM, Ferreira F (1990) Effects offoveal processing difficulty on the perceptualspan in reading: Implications for attentionand eye movement control. Journal of Ex-perimental Psychology: Learning, Memory,& Cognition 16(3):417–429.

[15] Henderson JM, Pollatsek A, Rayner K (1987)Effects of foveal priming and extrafoveal pre-view on object identification. Journal of Ex-perimental Psychology: Human Perception& Performance 13(3):449–463.

[16] Lavie N, Hirst A, de Fockert JW, Viding E(2004) Load theory of selective attention andcognitive control. Journal of ExperimentalPsychology: General 133(3):339–354.

[17] Hooge ITC, Erkelens CJ (1999) Peripheralvision and oculomotor control during visualsearch. Vision Research 39(8):1567–1575.

[18] Shen J, Reingold EM, Pomplun M, WilliamsDE, et al. (2003) Saccadic selectivity dur-ing visual search: The influence of centralprocessing difficulty, eds Hyona J, Radach R,Deubel H (Elsevier), pp 65–88.

[19] Morrison RE (1984) Manipulation of stim-ulus onset delay in reading: Evidence for

parallel programming of saccades. Journalof Experimental Psychology: Human Percep-tion & Performance 10(5):667–682.

[20] Reichle E, Pollatsek A, Fisher D, Rayner K(1998) Toward a model of eye movementcontrol in reading. Psychological Review105(1):125–157.

[21] Engbert R, Nuthmann A, Richter EM, KlieglR (2005) SWIFT: A dynamical model of sac-cade generation during reading. Psychologi-cal Review 112(4):777–813.

[22] Salvucci DD (2001) An integrated model ofeye movements and visual encoding. Cogni-tive Systems Research 1(4):201–220.

[23] Reichle ED, Pollatsek A, Rayner K (2012)Using EZ-reader to simulate eye movementsin nonreading tasks: A unified framework forunderstanding the eye-mind link. Psycholog-ical Review 119(1):155–185.

[24] Ahumada AJ (2002) Classification imageweights and internal noise level estimation.Journal of Vision 2(1):8.

[25] Victor JD (2005) Analyzing receptive fields,classification images and functional images:challenges with opportunities for synergy.Nature Neuroscience 8(12):1651–1656.

[26] Murray RF (2011) Classification images: Areview. Journal of Vision 11(5):2.

[27] Ludwig CJH, Gilchrist ID, McSorley E, Bad-deley RJ (2005) The temporal impulse re-sponse underlying saccadic decisions. Jour-nal of Neuroscience 25(43):9907–9912.

[28] Green DM, Swets JA (1966) Signal Detec-tion Theory and Psychophysics (New York:Wiley).

[29] DeLong ER, DeLong DM, Clarke-Pearson,DL (1988) Comparing the areas under two or


more correlated receiver operating character-istic curves: a nonparametric approach. Bio-metrics 44(3):837–845.

[30] Tolhurst DJ, Movshon JA, Dean A (1983)The statistical reliability of signals in singleneurons in cat and monkey visual cortex. Vi-sion Research 23(8):775–785.

[31] Britten KH, Shadlen MN, Newsome WT,Movshon JA (1992) The analysis of visualmotion: A comparison of neuronal and psy-chophysical performance. Journal of Neuro-science 12(12):4747–4765.

[32] Thompson KG, Hanes DP, Bichot NP, SchallJD (1996) Perceptual and motor pro-cessing stages identified in the activity ofmacaque frontal eye field neurons during vi-sual search. Journal of Neurophysiology76(6):4040–4055.

[33] Hooge ITC, Beintema JA, van den Berg A(1999) Visual search of heading direction.Experimental Brain Research 129(4):615–628.

[34] Pointer J, Hess R (1989) The contrast sensi-tivity gradient across the human visual field:With emphasis on the low spatial frequencyrange. Vision Research 29(9):1133–1151.

[35] Abbey CK, Eckstein MP (2006) Classifica-tion images for detection, contrast discrimi-nation, and identification tasks with a com-mon ideal observer. Journal of Vision 6(4):4.

[36] Ludwig CJH, Eckstein MP, Beutter BR(2007) Limited flexibility in the filter under-lying saccadic targeting. Vision Research47(2):280–288.

[37] Kass RE, Raftery AE (1995) Bayes factors.Journal of the American Statistical Associa-tion 90(430):773–795.

[38] Rouder JN, Speckman PL, Sun D, MoreyRD, Iverson G (2009) Bayesian t tests foraccepting and rejecting the null hypothesis.Psychonomic bulletin & review 16(2):225–237.

[39] Posner MI (1980) Orienting of attention. TheQuarterly Journal of Experimental Psychol-ogy 32(1):3–25.

[40] Treisman AM, Gelade G (1980) A feature-integration theory of attention. CognitivePsychology 12(1):97–136.

[41] Gold JI, Shadlen MN (2001) Neural com-putations that underlie decisions about sen-sory stimuli. Trends in Cognitive Sciences5(1):10–16.

[42] Smith PL, Ratcliff R (2004) Psychology andneurobiology of simple decisions. Trends inNeurosciences 27(3):161–168.

[43] Ratcliff R (1978) Theory of memory re-trieval. Psychological Review 85(2):59–108.

[44] Ratcliff R (2006) Modeling response signaland response time data. Cognitive Psychol-ogy 53(3):195–237.

[45] Ludwig CJ, Davies JR (2011) Estimat-ing the growth of internal evidence guidingperceptual decisions. Cognitive Psychology63(2):61 – 92.

[46] Beutter BR, Eckstein MP, Stone LS (2003)Saccadic and perceptual performance in vi-sual search tasks I. Contrast detection anddiscrimination. Journal of the Optical Soci-ety of America A - Optics, Image Science, andVision 20(7):1341–1355.

[47] Ludwig CJH (2009) Temporal integration ofsensory evidence for saccade target selection.Vision Research 49(23):2764–2773.


[48] Morrone M, Denti V, Spinelli D (2002) Colorand luminance contrasts attract independentattention. Current Biology 12(13):1134–1137.

[49] VanRullen R, Reddy L, Koch C (2004) Vi-sual search and dual tasks reveal two distinctattentional resources. Journal of CognitiveNeuroscience 16(1):4–14.

[50] Eckstein MP (2011) Visual search: A retro-spective. Journal of Vision 11(5):14.

[51] Pelli DG, Palomares M, Majaj NJ (2004)Crowding is unlike ordinary masking: Dis-tinguishing feature integration from detec-tion. Journal of vision 4(12):12.

[52] Levi DM (2008) Crowding-an essentialbottleneck for object recognition: A mini-review. Vision Research 48(5):635.

[53] Zelinsky GJ, Peng Y, Berg AC, Samaras D(2013) Modeling guidance and recognitionin categorical search: Bridging human andcomputer object detection. Journal of Vision13(3):30.

[54] Rayner K (1998) Eye movements in readingand information processing: 20 years of re-search. Psychological Bulletin 124(3):372–422.

[55] Pelli DG, Burns CW, Farell B, Moore-PageDC (2006) Feature detection and letter iden-tification. Vision Research 46(28):4646–4674.

[56] Findlay JM, Gilchrist ID (2003) Active Vi-sion: The Psychology of Looking and Seeing(Oxford University Press).

[57] Marigold DS, Weerdesteyn V, Patla AE, Duy-sens J (2007) Keep looking ahead? re-direction of visual fixation does not alwaysoccur during an unpredictable obstacle avoid-ance task. Experimental Brain Research176(1):32–42.

[58] Thorpe SJ, Gegenfurtner KR, Fabre-ThorpeM, Bülthoff HH (2001) Detection of ani-mals in natural images using far peripheralvision. European Journal of Neuroscience14(5):869–876.

[59] Hazeltine E, Teague D, Ivry RB (2002)Simultaneous dual-task performance revealsparallel response selection after practice.Journal of Experimental Psychology: HumanPerception & Performance 28(3):527–545.

[60] Itti L, Koch C (2000) A saliency-basedsearch mechanism for overt and covert shiftsof visual attention. Vision Research 40(10–12):1489–1506.

[61] Najemnik J, Geisler WS (2005) Optimal eyemovement strategies in visual search. Nature434(7031):387–391.

[62] Torralba A, Oliva A, Castelhano MS, Hen-derson JM (2006) Contextual guidance ofeye movements and attention in real-worldscenes: the role of global features in objectsearch. Psychological review 113(4):766–786.

[63] Zelinsky GJ (2008) A theory of eye move-ments during target acquisition. Psychologi-cal review 115(4):787–835.

[64] Schall JD, Thompson KG (1999) Neuralselection and control of visually guided eyemovements. Annual Review of Neuroscience22:241–259.

[65] Brainard DH (1997) The psychophysics tool-box. Spatial Vision 10(4):433–436.

[66] Pelli DG (1997) The psychophysics toolbox.Spatial Vision 10(4):437–442.

[67] Kleiner M, Brainard D, Pelli D (2007)What’s new in Psychtoolbox-3? Perception36S.


[68] Macmillan N, Creelman C (2004) Detectiontheory: A user’s guide.

Supporting InformationFoveal analysis and peripheral selection during active visual

sampling

Casimir J.H. Ludwig and J. Rhys DaviesUniversity of Bristol, School of Experimental

Psychology, Bristol, UK

Miguel P. EcksteinUniversity of California Santa Barbara,

Department of Psychological and Brain Sciences,Santa Barbara, United States

Behavioural performance

The performance data are summarised in FigureS1. For the critical measures – same-different ac-curacy, foveal target tilt discrimination, peripheraltarget tilt discrimination, and peripheral selection –the confidence intervals do not include chance per-formance (in fact, the conditional accuracy mea-sures are also reliably above chance, but this resultis less critical). Peripheral target tilt discriminationis somewhat lower than foveal target tilt discrimi-nation (compare 2nd and 3rd bars). There are twoexplanations for this effect: (i) the observer mighthave fixated the wrong peripheral pattern; (ii) evenif the target was fixated, the initial fixation durationwas longer than half the total trial duration and lesstime would have been available for processing thetarget.

We can assess the first of these two factors byexamining peripheral target tilt discrimination con-ditional upon saccade accuracy. When restrictingthe analysis to just those trials in which the tar-get was fixated first, performance is a little moreaccurate (a difference of about 3 - 6%; 4th bar).However, when taking only those trials in whichone of the non-targets was fixated (without subse-quent movements to the target), performance dropsclose to chance (about 54 - 56% correct; 5th bar).Indeed, on these trials, observers were more likelyto report the orientation of the non-target they erro-neously fixated (about 58 - 66%; 6th bar). It shouldbe pointed out that observers may not necessarily

realise they have fixated the wrong item and there-fore report the orientation of what they believe isthe target.

same - different

fovealtarget tilt

peritarget tilt

peri tilt| correct move

peri tilt| error move

non-tartilt | error

move

periselection

0.33

0.5

0.75

1

prop

ortio

n re

spon

ses

Figure S1. Overall performance accuracy. Fromleft to right: same - different judgement; fovealtarget tilt discrimination; inferred peripheral targettilt discrimination, regardless of the saccade end-point; peripheral target tilt discrimination, given anaccurate target-directed movement; peripheral tar-get tilt discrimination, given an error movement toa non-target; inferred “non-target” tilt discrimina-tion, given a movement to that non-target; periph-eral target selection accuracy with the first saccade.Bars correspond to the means across all observers;the error bars show the 95% confidence intervals,corrected for between-subject variability; crossesindicate individual observers’ performance. Thewhite dashed lines indicate chance levels.


Peripheral selection integration windowderived from correct saccade trials

The integration windows for peripheral selec-tion shown in Figures 3 - 5 in the main text werederived from trials in which the first saccade wasdirected to a non-target. As such, we comparedthe contrast noise at the chosen non-target loca-tion with that at the ignored target location. Weselected error trials because these are most likelyto be diagnostic in this regard: Errors may be ex-pected to occur when the external noise acts to sub-stantially enhance the contrast of a non-target pat-tern and/or substantially reduce the contrast of theperipheral target. Correct saccades may of coursealso be driven (partly) by external noise, such aswhen a noise sample enhances the target contrasteven further away from the non-target contrast.However, even in the absence of such noise val-ues, the sensory evidence will typically point to thecorrect target. As such, we would not expect a par-ticularly strong relation between the noise and thesaccadic decisions on these trials.

To illustrate this point, Figure S2A shows theraw temporal classification images for peripheralselection for the same observer whose data wasshown in the bottom row of Figure 3 of the maintext. The two ignored non-targets now hover closeto zero, as might be expected: when the noiseis around zero (which it is by definition most ofthe time), the sensory evidence favours the target.There is a small effect of noise at the target loca-tion, in that noise early in the trial that elevates thetarget may contribute to the correct eye movementdecisions.

Figure S2B shows the classification accuracy forthe sample as a whole. The green functions are thesame as those illustrated in Figure 4 of the maintext, based on the more informative error saccades.The grey functions are derived from the correctmovements only. As expected, the noise is muchless predictive of these correct decisions, althoughthe temporal profile – insofar as there is one – isconsistent with that derived from the error trials.From this analysis we conclude that there is little

to be gained by considering the correct saccadesin the identification of the integration window forperipheral selection.

Random versus blocked variation in fovealprocessing load

Our sample of eight observers was split into twogroups. One group received the variation in fovealprocessing load (i.e. magnitude of the mean tiltoffset) randomly inter-mixed. The other group re-ceived this variation in a blocked manner (in oneexperimental session, the foveal load was low intwo blocks and high in another two blocks of tri-als). Our expectation was that in the blocked con-dition, we might see strategic adjustments in theuptake of foveal information, with potential in-teractions with the uptake of peripheral informa-tion. Figure S3 shows the temporal processingwindows, separately for the two groups and fovealload levels. As in Figure 5 in the main text, thefunctions for the two difficulty levels show a greatamount of overlap. This finding suggests thateven when foveal processing load is blocked, nostrategic adjustment in the processing window wasmade.

Peripheral processing is non-unitary

Both orientation and contrast are coded by earlyvisual mechanisms at the level of primary visualcortex. As a result, it may be that processing pe-ripheral contrast information automatically also in-volves extracting orientation information.

Moreover, in the Discussion of the main text weraise the possibility of a rapidly shifting serial at-tention mechanism. In particular, we describe howsuch a mechanism may give rise to temporal inte-gration windows that mimic genuine parallel pro-cessing of foveal and peripheral information. Suchan attentional spotlight is typically thought of asa unitary mechanism that is needed to bind differ-ent features within the focus of attention together[1]. Given the unitary nature of the mechanism,we would expect the uptake of contrast and orien-


0 150 300 450 600 750

−0.01

0

0.01

time from display onset (ms)

cont

rast

noi

se

display aligned

targetnon−target(s)

−400 −200 0 200 400

−0.01

0

0.01


saccade aligned

0 150 300 450 600 750

0.5

0.55

0.6

0.65


clas

sific

atio

n ac

cura

cy (a

rea

unde

r RO

C)

−400 −200 0 200 400

0.5

0.55

0.6

0.65


error saccadescorrect saccades

A

B

Figure S2. Noise classification analysis for periph-eral selection based on correct saccades only. A– Raw temporal classification images for one ob-server. The two non-targets were labelled 1 and 2,after rotating the configuration of three peripheralpatterns in a clockwise direction to coincide withthe “top” configuration shown in Figure 2 of themain text. Non-target 1 is then the “right-most”non-target (e.g. if the target was horizontal right,non-target 1 would be the top pattern). B – Noiseclassification accuracy for the total sample of 8 ob-servers, shown separately for error saccades (thesame green curves as in Figure 4 of the main text)and correct saccades (grey functions). The noisevalues from the two non-targets are pooled to-gether in the “ignored non-target” distribution forthe purpose of this analysis (i.e. this distributionwill have twice the number of values at each pointin time to the number of “chosen target” samples).The shaded regions around the functions show the95% confidence intervals across subjects. In thesaccade aligned plots (right-hand column), the av-erage saccade duration is shown by the grey shadedbox.

tation information from peripheral locations to gohand-in-hand.

For these reasons, we analysed the time courseof contrast and orientation processing in the pe-riphery, contingent upon different saccadic deci-sions. In these analyses we are really only inter-ested in the uptake of information before move-ment onset. As such, we only show the noise clas-sification accuracy aligned on movement onset.

Figure S4A shows the uptake of contrast infor-mation for peripheral selection (in green, as inmain Figure 4). In addition, we plot the uptake oftilt information from the peripheral target location,given accurate selection of the saccade target (inmagenta - replicated in all three panels). Note thatfor the peripheral target, we have to infer the ob-server’s tilt judgement from his/her perceived tiltof the foveal target and the overall same-differentjudgement.

Some tilt information from the peripheral targetis processed while the eyes are still focused on thefoveal target. The time course of the two functionssuggests that peripheral tilt information starts be-ing processed once the peripheral selection func-tion has reached its peak. This peak representsthe point in time that, on average, is most predic-tive of the ensuing saccadic decision. As such, itmay be considered as an index of the completionof saccade target selection. Target selection mayfacilitate visual processing of visual features fromthe future fixation position, consistent with pre-saccadic shifts of covert attention reported else-where [2, 3, 4]. After the saccade, processing of tiltat the peripheral target location is greatly enhancedbecause the pattern now falls onto the fovea.

Is the pre-saccadic processing of orientation in-formation specific to the (saccade) target locationor does it occur for all peripheral patterns in thebuild-up to a gaze shift? We address this ques-tion by looking at: (i) the uptake of tilt informationfrom the peripheral target location when a saccadeis generated to a different pattern (i.e. an “error”saccade); (ii) the uptake of tilt information from aperipheral non-target location, given an impendingerror saccade to that location.

Panels B and C show these comparisons. In


0 150 300 450 600 750

0.5

0.55

0.6

0.65

clas

sific

atio

n ac

cura

cy (a

rea

unde

r RO

C)

random

0 150 300 450 600 750

0.5

0.55

0.6

0.65blocked

0 150 300 450 600 750

0.5

0.55

0.6

0.65

time (ms) wrt display onset0 150 300 450 600 750

0.5

0.55

0.6

0.65


−400−200 0 200 400

0.5

0.55

0.6

0.65random

−400−200 0 200 400

0.5

0.55

0.6

0.65blocked

−400−200 0 200 400

0.5

0.55

0.6

0.65

time (ms) wrt saccade onset−400−200 0 200 400

0.5

0.55

0.6

0.65


foveal identification

peripheral selection


display aligned saccade aligned

Figure S3. Noise classification accuracy for foveal identification and peripheral selection for the twolevels of foveal processing load. Functions are averaged over the four observers in each group. In thesaccade aligned plots (3rd and 4th colunns), the duration of the saccade is shown by the shaded verticalbox. Shaded regions around the functions are 95% confidence intervals. These are larger than in Figures4 and 5 in the main text, because each function is now based on only half the subject sample.

panel B we show that no orientation informationfrom the peripheral target is processed before thesaccade when the movement is directed elsewhere.Indeed, the divergence of these curves about 175ms before movement onset may be taken as an es-timate of when the pre-saccadic facilitation at thefuture fixation position begins.

In Panel C we show the uptake of orientationinformation from a peripheral non-target location,when that location is about to be fixated. Notethat for this analysis, we take the inferred target-tiltjudgement and assess it against the true tilt offsetat the fixated non-target location. Responses thatare congruent with the tilt offset at that location areconsidered “correct” for the purpose of the noise-classification analysis (and incongruent responsesare treated as “errors”).

This final analysis also addresses a concern onemight have with the comparison between periph-

eral selection based on contrast and the uptake oftilt information at the saccade target location. Thetwo curves shown in panel A are based on differ-ent sub-sets of trials. That is, the function for pe-ripheral selection is based on error saccades, butthe function for peripheral tilt processing is basedon correct saccades. However, panel C demon-strates that the uptake of peripheral tilt informa-tion from the future fixation location is the samefor correct and error saccades (at least insofar asthe pre-saccadic facilitation is concerned).

These analyses demonstrate that the time coursefor processing contrast and orientation informationfrom the periphery was very different in the build-up to a gaze shift. Peripheral contrast informationis processed right from the start of a fixation fromall possible saccade target locations (Figure 3 inthe main text). However, orientation information isonly processed at the future fixation position, once


−400 −200 0 200 400

0.5

0.55

0.6

0.65

−400 −200 0 200 400

0.5

0.55

0.6

0.65

clas

sific

atio

n ac

cura

cy (a

rea

unde

r RO

C)

−400 −200 0 200 400

0.5

0.55

0.6

0.65


peripheral selectionper. tar. tilt | correct sac.

per. tar. tilt | correct sac.per. tar. tilt | error sac.

per. tar. tilt | correct sac.per. non−tar. tilt | error sac.

A

C

B

Figure S4. Noise classification accuracy for pe-ripheral selection and peripheral tilt discrimina-tion, contingent upon the accuracy of saccade tar-get selection. The functions are averaged acrossall eight observers. The vertical shaded box indi-cates the saccade duration. Shaded areas aroundthe functions correspond to 95% confidence inter-vals.

saccade target selection is at least partly complete.The differential uptake of contrast and orientationinformation argues strongly against a rapidly shift-ing unitary attention mechanism. These results areone reason why we question the utility of an atten-tional spotlight in accounting for behaviour in thepresent study.

References

[1] Treisman AM, Gelade G (1980) A feature-integration theory of attention. Cognitive Psy-chology 12:97–136.

[2] Kowler E, Anderson E, Dosher BA, Blaser E(1995) The Role of Attention in the Program-ming of Saccades. Vision Research 35:1897–1916.

[3] Deubel H, Schneider W (1996) Saccade tar-get selection and object recognition: Evidencefor a common attentional mechanism. VisionResearch 36:1827–1837.

[4] Rolfs M, Jonikaitis D, Deubel H, CavanaghP (2011) Predictive remapping of attentionacross eye movements. Nature Neuroscience

14:252–256.

Documents

Ludwig, C. J. H. , Davies, J. R., & Eckstein, MP. (2014 ... · Casimir J.H. Ludwig and J. Rhys Davies University of Bristol, School of Experimental Psychology, Bristol, UK Miguel