Activation of Lexical Assemblies is Reflected in Mouse Movements

Fermın Moscoso del Prado Martın (fermosc@gmail.com)Laboratoire de Psychologie Cognitive, CNRS & Aix-Marseille Universite, Marseille, France &

Laboratoire Dynamique du Langage, CNRS & Universite de Lyon, Lyon, France &Institut Rhone-Alpin des Systemes Complexes (IXXI), Lyon, France

Laurie B. Feldman (lfeldman@albany.edu)SUNY – The University at Albany, NY, USA &

Haskins Laboratories, New Haven, CT, USA

Rick Dale (rdale@ucmerced.edu)University of California, Merced, CA, USA

AbstractRecent studies have shown that cognitive processing isreflected in a change of the entropy of behavioral mea-sures. Novel in the present study is evidence that lexicalprocessing is reflected in the variability (i.e., entropy) ofmotor responses. We performed a lexical decision task,where the subjects responded by moving their mouse tolocations designated as ”word” or ”nonword”. We ana-lyzed the entropy of the sequences of cursor velocities inindividual trials. We found that the movement entropyof correct trials corresponding to words was reduced rel-ative to the movement entropy of trials corresponding topseudo-words. However, this pattern was reversed in er-roneous responses, in which case movement entropy towords was higher than that to pseudo-words. Impor-tantly, these effects cannot be explained just in termsof reaction times. We argue that the decreases in move-ment entropies are a direct consequence of the activationof neural assemblies representing words.Keywords: Entropy, Lexical Decision, MechanicalTurk, Mouse Tracking, Neural Assemblies

IntroductionConverging sources of neurophysiological evidence sug-gest that words are represented by broadly distributednetworks of strongly associated cortical neurons, whichare referred to as neural assemblies (cf. Pulvermuller,1999). These assemblies encode the formal (orthographicand phonetic), pragmatic, and semantic properties of aword. On presentation of a word, these assemblies fire insynchrony and reverberate at very high frequencies. Thishigh-frequency reverberation is observed as an increaseof power in the gamma band (frequencies greater than40 kHz.) of EEG and MEG signals. The activation ofa neural assembly would make the neural activity dur-ing lexical processing more predictable, that is to say,it is easier to predict where and when neural activitywill take place when an assembly is reverberating thanwhen it is not. In other words, the disorder or entropy(Shannon, 1948) of cortical activity should decrease dur-ing lexical processing. Such effects of decreased corticalentropy during cognitive processing have been measuredusing spectral entropy (cf., Rezek & Roberts, 1998).

In a recent study, Moscoso del Prado (2011) has shownthat the entropy of the distribution of reaction times re-lates to the amount of cognitive processing, in a more

powerful way than do traditional measures of reactiontime. It is argued that the entropy of behavioral re-sponses is a reflection of the entropy of the cognitive sys-tem, including of course cortical neurons. If this is indeedthe case, one would expect that the activation of a neuralassembly should also be reflected in the entropy of behav-ioral responses. Therefore, one would predict that theentropy of behavioral responses should decrease whenprocessing words relative to processing pseudo-words.Furthermore, if –as maintained by Pulvermuller (1999)–lexical assemblies encompass neurons from a multitudeof cortical areas (including the motor and perceptual sys-tems), it is to be expected that this entropy reduction isobservable even when the behavioral response itself haslittle to do with lexical processing.

In this study we test the prediction that lexical pro-cessing should result in a reduced entropy of behavioralresponses. For this purpose, we use a visual lexical lexi-cal decision experiment in which the response is providedby moving the cursor using a mouse. Participants are in-structed to move the mouse to click on a WORD button,when the presented stimulus is a word, and to move toand click on a NON-WORD button when the stimulusis a plausible English pseudo-word. Correct responses insuch a task indicate that some form of lexical represen-tation has been activated. Lexical decision responsesare sensitive to variety of formal and semantic prop-erties of a word; (cf., Baayen, Feldman, & Schreuder,2006), hence one can expect that they elicit the acti-vation of the corresponding assemblies. On the otherhand, in cases where participants make errors, one couldinterpret these errors as being caused by the failure toactivate the corresponding word assembly (i.e., misses),or by inappropiate activation of a lexical assembly by apseudo-word (i.e., false alarms). We examine the entropyof the trajectory generated by the mouse for each type oftrial. Following the reasoning above, we predict reducedvalues of the movement entropy when processing wordsrelative to pseudo-words for correct responses, and theexact reverse pattern for the incorrect responses.

Table 1: Stimuli chosen from Balota et al. (2000).

High-frequency words Low-frequency words Pseudo-wordsbeach apple bealtcame blast blaspdark bolo cadefaster chomp darphall chump mankleast fogy martellist glass milt

main hack mokemarket morass pamtlemember pamper pilk

miss pill semmore pomp sibname sip soptpower sobs stimp

see sock sumpelstill stink trafe

these summit twytime wade waimtrain weed weebwest yelp yuke

Materials and Methods

Participants133 participants recruited online through Amazon Me-chanical Turk partook in the study for monetary com-pensation. Recruitment was restricted to the UnitedStates.

MaterialsA set of words and non-words was selected from a studyby Balota, Law, and Zevin (2000). Four practice items(excluded from the analyses) were presented as the firstfour trials (2 word, 2 pseudo-word). The were followedby 20 high-frequency words, 20 low-frequency words, and20 non-words. Stimuli are shown in Table 1. As de-scribed by Balota et al., high-frequency words occurredat a median frequency of about 350 times per millionwords in text; low-frequency words occurred approx-imately 6 times per million words. The words wereabout equal in terms of length and orthographic neigh-bors. Plausible pseudo-words were chosen by randomlyselecting a subset of 20 of the non-words used in Balotaet al. (2000). After piloting with the task online, wechose to limit the set of pseudo-words to keep the exper-iment brief. A web-based interface was programmed inAdobe Flash, and participants responded to these stim-uli by clicking WORD or NON-WORD buttons in theirbrowser.

ProcedureAfter reading a brief description of the task, partici-pants were presented with a window in their browserthat contained WORD and NON-WORD buttons in the

top left and top right of their browser window (random-ized across participants, consistent within). A fixationcircle appeared at the bottom center of the window, andtrials were initiated by clicking this circle. Upon click-ing this circle, a stimulus item appeared near it and par-ticipants were instructed to respond WORD or NON-WORD by clicking the corresponding button with theircomputer mouse. After clicking the button, the fixationcircle reappeared and participants began the next trial.After doing 4 practice trials, participants completed the60 word and non-word trials in random order, and partic-ipants carried out the task with their mouse and AdobeFlash in their browser. The task required approximately10-15 minutes.

Data and Analysis

Preprocessing As participants responded, themouse-cursor trajectory was sampled at approximately40 Hz. By initiating each trial at the bottom centerof the screen, each trajectory reflected a consistentcenter-to-left or center-to-right movement to one of thebuttons (akin to Spivey, Grosjean, & Knoblich, 2005;Dale, Kehoe, & Spivey, 2007). These were saved in textfiles and subjected to analysis by converting the trajec-tories into velocity profiles in the manner described inMcKinstry, Dale, and Spivey (2008): We divided theaverage movement over a six-sample window based onthe average amount of time elapsed. As described inMcKinstry et al., the moving average window smoothsover noise inherent in the mouse sampling. Finally, thesequences of zero velocities at the beginnings and endsof each sequence were removed. This new velocity timeseries now reflects the various moments of increasingand decreasing rate of movement that a subject carriedout during a trial, and thus may serve as a window ontothe complexity of the unfolding cognitive process (seeFreeman, Dale, & Farmer, 2011, for a review). Notethat the raw sequences of mouse positions would beproblematic for the entropy computations below; eachraw position corresponds to the previous one, plus asmall increment, resulting in strongly autocorrelatedsequences. In this respect, the sequence of velocities (thetemporal derivatives of the positions), also correspondbetter to the actual movements.

Entropy Estimation For each trial we estimated theentropy of the corresponding sequence of velocities, usinga technique akin to that described in Moscoso del Prado(2011). First, a series of breakpoints was defined byapplying the Sturges algorithm (Sturges, 1926) to theconcatenation of all velocity sequences across all trialsof all subjects. This resulted in fifteen bins, each havinga width of 500 pixels/s., jointly spanning the range be-tween 0 pixels/s. and 7500 pixels/s. For the sequence ofvelocities from each individual trial, we estimated a fre-quency histogram using these common break-points for

all trials. As the individual velocity sequences are farshorter than the overall concatenated one, the resultinghistograms contain many empty bins. To correct thisproblem, the individual histograms were smoothed usingthe James-Stein shrinkage method (Hausser & Strimmer,2009). This histogram smoother produces accurate esti-mates of the entropy even in cases that suffer from severeunder-sampling. For each individual trial, the smoothedhistograms provided fifteen values p1, p2, . . . , p15, corre-sponding to the estimated probabilities of the velocityfalling in the corresponding bins. From these probabili-ties, for each trial, we estimated the entropy of the his-togram using its classical expression (Shannon, 1948)

H = −15∑

i=1

pi log15 pi.

Notice that, by using logarithms to the base of the totalnumber of bins per histogram (15), the entropy is thenexpressed in relative units with a minimum of zero (con-stant; all values falling in the same bin) and a maximumvalue of one (full chaos; values evenly distributed acrossbins). In this way, the entropy values can be interpretedas a ‘proportion’ relative to the maximum possible dis-order.

Regression Analyses The accuracy of the entropyestimates depends on the length of the sequence on whicheach was computed. The shorter a sequence is, the moreits entropy is underestimated. This introduces a con-found to the interpretation of any possible effects onthe entropies. The sequence durations themselves arecorrelated with the duration of the movement that de-fined the mouse trajectory (the time from start to endof the movement), and their latencies (the lag betweenstimulus presentation and initiation of movement). It islong-known that reaction times in lexical decision (cor-responding in this case to the sum of the response la-tency and the movement duration) are strongly corre-lated with both the frequency and the lexicality of thepresented stimuli. Therefore, in order to ensure that anyeffects we may observe are not confounded with moretraditional effects on reaction times, but capture insteadgenuine effects on motor behavior over and above anycontribution of reaction time and sequence length. Theabove requires partialling out the contributions of se-quence length, movement duration, and latency beforeassessing the significance of any effect.

As is illustrated in Figure 1, the length, duration, andlatency measures have strong log linear relations to theentropy (and further among themselves), apparently in-dependently of the experimental condition. Includingthese three measures on a plain regression model couldresult in an unacceptable level of collinearity, that wouldmake it impossible to interpret any results. To circum-vent this problem, we orthogonalized these three compo-

nents by means of a principal component analysis (PCA)of the three inter-correlated measures (on logarithmicscale). This produces a rotation into three new compo-nents that are fully orthogonal (i.e., uncorrelated), butthat still account for all the information that was con-tained in the original measures.

In order to estimate the effect of frequency and lexi-cality of the items, and the correctness of the responses,on the entropies of the movements, we fitted a regres-sion model with the estimated entropy of the sequenceof velocities as the dependent factor, and experimentalcondition (high-frequency word vs. low-frequency wordvs. pseudo-word) and response type (correct vs. er-ror), and their interaction, as dependent variables. Todiscount the possible effects of the reaction time and se-quence length variables, the three principal componentsdescribed above were included as additional co-variatesin the regression. As illustrated by the smoothers in Fig-ure 1, the logarithmic reaction time and sequence lengthvalues are non-linearly related to the entropy estimates.As PCA only performs a linear transformation (ie., arotation) on the original variables, it is clear that theprincipal components will themselves be non-linearly re-lated to the entropy values. In order to also partial outany possible contribution of these non-linearities, the re-gression was fitted using the Generalized Additive Model(GAM) technique (cf., Hastie & Tibshirani, 1990) usingquadratically penalized regression splines (Wood, 2004)to model the non-linearities on the effects of the threeprincipal components.

Finally, one must take into account that there is con-siderable heterogeneity across subjects and items in thisdataset. Most obviously, the sequences were collectedfrom different subjects, who were performing the exper-iment on different equipment. The varying degree of‘shakiness’ of different individuals, and the different sen-sitivities and configurations of different equipment willconsistently affect the entropy estimates (i.e., entropyand variability are intimately related). Furthermore, theexperimental design is extremely uneven. The lexicalityand frequency properties of items are well-known to in-fluence the number of errors that they will elicit, andsome subjects are more error prone than others. Thiswill result in substantially different numbers of pointsin each experimental cell (e.g., error responses high-frequency words will be less numerous than error re-sponses to low-frequency words, correct responses willbe more abundant than errors across the board, etc.).Similar issues will affect the relation between the reac-tion time measures and the entropy estimates. Theseproblems require the explicit consideration of randomeffects for each subject, item, and for the shapes of thenon-linear components, using the mixed-effect modelingmethodology (Bates, 2005). Therefore the models werefitted combining non-linear GAM spline effects for the

Rel

ativ

e E

ntro

py

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

Length of Velocity Sequence

100.5 101 101.5 102 102.5 103

Duration of Movement

102 102.5 103 103.5 104 104.5

Latency

101.5 102 102.5 103 103.5 104 104.5

ConditionNW

HF

LF

Figure 1: Correlations between the estimated entropies, and the length (in number of points) of the correspondingvelocity sequence (left), the duration (in ms.) of the movement (middle), and the response latency (in ms.; right).The black lines are non-parametric smoothers, and the shading denotes their standard error. Notice the logarithmicscale on the horizontal axes.

principal components while explicitly considering ran-dom effects of subject identity, item identity, and theshapes of the nonlinear splines. This was achieved usingthe Generalized Additive Mixed-effect Model (GAMM;Wood, 2006), as implemented in the R package gamm4.The final random effect structure was chosen by com-paring the performance of different random effect struc-tures (using Schwarz’s Bayesian Information Criterion;Schwarz, 1978). Model criticism was performed on thefinal model by ascertaining the normality of the modelresiduals, and that these residuals were not correlatedwith the dependent variable.

Results and Discussion

Figure 2 summarizes the mean effects of the sequencelength and reaction time variables (through the corre-sponding principal compenents) that were partialled outby the GAM components of the models. The effects ofthe three principal components were significant in themodel (first principal component: F [7.294, 7814.896] =275.13.54, p < .0001; second principal component:F [6.449, 7814.896] = 371.49, p < .0001; third principalcomponent: F [7.361, 7814.896] = 52.86, p < .0001).1

Discounting the contribution of the reaction time andsequence length variables, there are still significant ef-fects of the fixed effect predictors. We observed signif-icant main effects of the frequency/lexicality condition(F [2, 7814.896] = 108.72, p < .0001), of the correctnessof the response (F [1, 7814.896] = 41.93, p < .0001), aswell as of their two-way interaction (F [2, 7814.896] =

1The non-integer degrees of freedom are the result of theapproximation of degrees of freedom of the spline componentsthat is performed in the GAMM method.

50.01, p < .0001). The effects of these variables on theestimated entropies are summarized by Figure 3.

Consider first the values of the entropies for the cor-rect responses. As was predicted, there is a signifi-cant increase in the movement entropies for the pseudo-words, with respect to both high frequency words andlow-frequency words. We interpret this increase as a re-flection of the reduction in the overall entropy of thecognitive system that results from the more orderly neu-ral activity caused by the activation of the cortical as-sembly that represents words. Notice, however, thatthis increase in activity is roughly equal with respectto high- and low-frequency words, which do not differsignificantly in the entropy they elicit (cf., strongly over-lapping standard errors in the plot).

On the other hand, erroneous responses elicit an al-most perfectly reversed pattern with respect to the cor-rect ones. In this case, it is the responses to the pseudo-words that show the lower entropy, roughly the same asthat elicited by correct responses to words. In contrast,both the high and the low frequency existing words elicitentropies that are more or less equal to the entropies ofcorrect responses to pseudo-words. We interpret thispattern as reflecting the incorrect activation of a lexicalassembly for incorrect responses to pseudo-words, andthe lack of activation of the lexical assembly in the in-correct responses to existing words. In this case, dueto the extremely low number of incorrect responses tohigh frequency words (.42%, versus 12.18% for the low-frequency words, and 11.60% for the pseudo-words), thehigh standard error is very high in these items. Stillthe differences between words and pseudo-words are veryclear, and the estimated values are roughly equal for

-6 -4 -2 0 2

-0.5

-0.4

-0.3

-0.2

-0.1

0.0

0.1

PC1

Ent

ropy

of V

eloc

ities

-2 0 2 4-0.5

-0.4

-0.3

-0.2

-0.1

0.0

0.1

PC2

Ent

ropy

of V

eloc

ities

-1.0 -0.5 0.0 0.5 1.0 1.5

-0.5

-0.4

-0.3

-0.2

-0.1

0.0

0.1

PC3

Ent

ropy

of V

eloc

ities

Figure 2: Effects on the relative entropy of the three principal components as estimated by the GAMM model. Theshaded areas plot the estimated standard errors. The rugs at the bottom of the plots represent the distribution ofvalues of each principal component.

both high-frequency and low-frequency words.The current results are important from both method-

ological and theoretical perspectives. On the method-ological side, the entropy of the sequence of veloci-ties offers a way to summarize a movement trajec-tory (in mouse tracking experiments, but equally ap-plicable to eye-tracking and other movement trackingparadigms) in a single informative number. As discussedin Moscoso del Prado (2011), the entropy measure in-dexes the state of the system that produced the trajec-tories (i.e., the cognitive system), and is related to thedegree of energy consumption in the system. Showingthat the effects of the movement patterns are evidenteven when the contribution of reaction time and latencymeasures has been discounted, highlights that movementtracking paradigms offer additional insights into cogni-tive processing over and above what can be found byanalyzing reaction time data alone. Notice that, al-though reaction time distributions for correct and in-correct responses, as well as for lexical and non-lexicalitems, are markedly different (cf., Ratcliff, Gomez, &McKoon, 2004), the entropy patterns after partiallingout time related variables show a consistent pattern,clearly suggestive of the activation (or lack thereof) oflexical representations, symetrically between correct andincorrect responses, or between words and pseudo-words.

One the theoretical side, with respect to lexical pro-cessing, these results are consistent with the hypothesisthat words are represented by widely distributed neuralassemblies, which fire immediately after visual or audi-tory presentation of a word (cf., Pulvermuller, 1999). AsPulvermuller proposes, these assemblies are widely dis-tributed across the cortical surface, rather than encap-sulated in a more or less modular way. The activation

of a neural assembly elicits a marked increased of thegamma band activity observed in electro-physiologicaland magneto-physiological studies, and such increasesare indeed observed in words in relation to pseudo-words(cf., Pulvermuller, 1999). Such an increase in the pro-portion of power in the gamma band would decrease theentropy of neural activity across the whole cortex (as ismeasured for instance in the resulting decrease in thespectral entropy of EEG or MEG signals; cf., Rezek &Roberts, 1998), and should therefore be observable indecreases in the entropy across the system, such as thedecreased entropy of motor activity that we observedhere. However, some have argued that what appears asan increase in gamma band activity for words in rela-tion to pseudo-words is in fact as a temporal delay inthe onset of the gamma activity for pseudo-words (e.g.,Urbach, Davidson, & Drake, 1999). Our results are notambiguous in this respect. The decrease in the entropyof the motor responses is observable for words in correctresponses. However, the decrease is also observable forthe pseudo-words in the incorrect responses, and thereis no evident way in which the reversed delay in gammaactivity could be justified in this case.

Our results are in line with the proposal ofMoscoso del Prado (2011), that the entropy of behav-ioral responses reflects the entropy, and ultimately theenergy consumption, of the cognitive system. Addi-tional evidence supporting this interpretation has alsobeen provided by Stephen and his co-workers (Stephen,Boncoddo, Magnuson, & Dixon, 2009; Stephen & Dixon,2009; Stephen, Dixon, & Isenhower, 2009). In problem-solving tasks they found that an entropy-based measureof eye-movement reflects the point at which the strategyhas been found by the participants. As in our case, the

Ent

ropy

of V

eloc

ities

0.55

0.56

0.57

0.58

0.59

0.60

0.61

0.62

High frequency Low frequency Pseudo-word

ResponseError

Correct

Figure 3: Mean values of the movement entropy foreach combination of experimental condition and correct-ness of the response as estimated by the GAMM model(taking into account the additional contributions of thespline effects of Figure 2 and the random effect of indi-vidual subjects and items). The error bars denote theestimated standard errors.

entropy of eye-movements (another motor behavior) re-flects the organization of higher cognitive functions. Weconcur with Stephen and his colleagues in that the non-linear properties of behavioral responses are informativeabout the dynamics of cognition.

References

Baayen, R. H., Feldman, L. B., & Schreuder, R. (2006).Morphological influences on the recognition of mono-syllabic monomorphemic words. Journal of Memory& Language, 55 , 290-313.

Balota, D. A., Law, M. B., & Zevin, J. D. (2000). Theattentional control of lexical processing pathways: Re-versing the word frequency effect. Memory & Cogni-tion, 28 , 1081–1089.

Bates, D. M. (2005). Fitting linear mixed models in R.R News, 5 , 27–30.

Dale, R., Kehoe, C., & Spivey, M. J. (2007). Graded mo-tor responses in the time course of categorizing atypi-cal exemplars. Memory & Cognition, 35 , 15–28.

Freeman, J. B., Dale, R., & Farmer, T. A. (2011). Handin motion reveals mind in motion. Frontiers in Psy-chology , 2 , 59.

Hastie, T. J., & Tibshirani, R. J. (1990). Generalizedadditive models. Boca Raton, FL: Chapman & Hall /CRC Monographs on Statistics & Applied Probability.

Hausser, J., & Strimmer, K. (2009). Entropy infer-ence and the James-Stein estimator, with application

to nonlinear gene association networks. Journal of Ma-chine Learning Research, 10 , 1469-01484.

McKinstry, C., Dale, R., & Spivey, M. J. (2008). Contin-uous attraction toward phonological competitors. Psy-chological Science, 19 , 22-24.

Moscoso del Prado, F. (2011). Macroscopic thermo-dynamics of reaction times. Journal of MathematicalPsychology , 55 , 302–319.

Pulvermuller, F. (1999). Words in the brain’s language(with discussion). Behavioral and Brain Sciences, 22 ,253–336.

Ratcliff, R., Gomez, P., & McKoon, G. (2004). A diffu-sion model account of the lexical decision task. Psy-chological Review , 111 , 159182.

Rezek, I. A., & Roberts, S. J. (1998). Stochasticcomplexity measures for physiological signal analysis.IEEE Transactions on Biomedical Engineering , 45 ,1186–1191.

Schwarz, G. (1978). Estimating the dimension of amodel. Annals of Statistics, 6 , 461–464.

Shannon, C. E. (1948). A mathematical theory of com-munication. Bell System Technical Journal , 27 , 379–423, 623–656.

Spivey, M. J., Grosjean, M., & Knoblich, G. (2005).Continuous attraction toward phonological competi-tors. Proceedings of the National Academy of SciencesU.S.A., 102 , 10393–10398.

Stephen, D. G., Boncoddo, R. A., Magnuson, J. S., &Dixon, J. A. (2009). The dynamics of insight: Math-ematical discovery as a phase transition. Memory &Cognition, 37 , 1132—1149.

Stephen, D. G., & Dixon, J. A. (2009). The self-organization of insight: Entropy and power laws inproblem solving. The Journal of Problem Solving , 2 ,72–101.

Stephen, D. G., Dixon, J. A., & Isenhower, R. W. (2009).Dynamics of representational change: Entropy, action,and cognition. Journal of Experimental Psychology:Human Perception and Performance, 35 , 1811–1832.

Sturges, H. A. (1926). The choice of a class interval.Journal of the American Statistical Association, 21 ,65–66.

Urbach, T. P., Davidson, R. E., & Drake, R. M. (1999).Gamma band suppression by pseudowords: Evidencefor lexical cell assemblies? Behavioral and Brain Sci-ences, 22 , 305–306.

Wood, S. N. (2004). Stable and efficient multiplesmoothing parameter estimation for generalized addi-tive models. Journal of the American Statistical As-sociation, 99 , 673–686.

Wood, S. N. (2006). Low rank scale invariant tensorproduct smooths for generalized additive mixed mod-els. Biometrics, 62 , 1025–1036.