Embed Size (px)
Citation preview
Neuroanatomical Correlates of Perceived Usability
Chi Thanh Vi
SCHI Lab
University of Sussex, UK
Kasper Hornbæk
University of Copenhagen,
Denmark
Sriram Subramanian
Interact Lab
University of Sussex, UK
Figure 1. Some key regions of brain activity (in white circles) for assessment of perceived usability (left to right): left medial frontal
gyrus, left superior frontal gyrus, right superior frontal gyrus, right precentral gyrus, left claustrum, and left putamen.
ABSTRACT
Usability has a distinct subjective component, yet
surprisingly little is known about its neural basis and relation
to the neuroanatomy of aesthetics. To begin closing this gap,
we conducted two functional magnetic resonance imaging
studies in which participants were shown static webpages (in
the first study) and videos of interaction with webpages (in
the second study). The webpages were controlled so as to
exhibit high and low levels of perceived usability and
perceived aesthetics. Our results show unique links between
perceived usability and brain areas involved in functions such
as emotional processing (left fusiform gyrus, superior frontal
gyrus), anticipation of physical interaction (precentral gyrus),
task intention (anterior cingulate cortex), and linguistic
processing (medial and bilateral superior frontal gyri). We
use these findings to discuss the brain correlates of perceived
usability and the use of fMRI for usability evaluation and for
generating new user experiences.
Author Keywords
User Experience, Aesthetics, Usability, fMRI
ACM Classification Keywords
H.5.2 [Information interfaces and presentation]: User
Interfaces - Graphical user interfaces;
INTRODUCTION Usability is a key concept in human-computer interaction
(HCI) that concerns the ease, effectiveness, and satisfaction
with which users achieve their goals when interacting with
computers. Usability can be measured both objectively (e.g.,
task completion time) and subjectively (e.g., asking users
about their satisfaction) [43]. Such measures are important
for evaluating computer systems and driving HCI research.
For instance, improved usability increases the return on
investment for information technology [53] and poor
usability wastes billions of dollars every year [11].
The construct of usability, however, raises many open
questions. For instance, the ISO 9241 standard names three
aspects of usability, but others include different dimensions
[40, 68, 79]. Thus, usability is not a clear construct.
Moreover, while the relationship between usability and
aesthetic appeal has been much researched (e.g., [7, 57, 58,
82]), one key question that remains is whether what is
beautiful is also perceived as usable [57, 82], that is, whether
and how immediate impressions of aesthetics influence
subsequent performance/ assessment of usability.
We present an exploratory study of the neuroanatomical
correlates of usability using functional magnetic resonance
imaging (fMRI). The key motivation is to fill the gap in
research on the neural basis of usability, and its similarities
and differences to aesthetics. The study is exploratory and we
hold no strong a priori hypotheses because (a) no prior work
has used fMRI to investigate the neural basis of usability and
(b) the literature on usability rarely hypothesize about
relations or overlap among constructs.
The goal of this work is to link different components of
usability to particular brain areas (and their underlying
cognitive functions). We do so by comparing differences in
brain activation during the judgment of aesthetics and
usability, so as to identify the overlapping and uniquely
activated brain areas and discuss associated cognitive
functions. This allows us to make three contributions:
We identify brain areas specific to usability and areas
shared with aesthetics;
We analyze usability theory based on an interpretation of
the function of those brain areas;
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for
components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to
post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from [email protected]. UIST 2017, October 22–25, 2017, Québec City, QC, Canada.
© 2017 ACM. ISBN 978-1-4503-4981-9/17/10...$15.00.
DOI: https://doi.org/10.1145/3126594.3126657
We outline uses of fMRI for usability evaluation and for
generating new user experiences.
RELATED WORK
Here, we will first discuss the construct of perceived usability
and then review the principles of fMRI. Then we will
highlight the lack of earlier fMRI studies on usability and
describe the benefits of this line of work.
Perceived Usability
Usability as a construct has been widely accepted and
disseminated in the HCI field. According to ISO [1],
usability concerns the effectiveness, efficiency, and
satisfaction with which users can achieve their goals when
interacting with computers. Recently, there has been a surge
of interest in user experience, bringing a focus on temporality
and concepts such as the affective and the hedonic [38]. In
addition, ISO [1] defines user experience as "a person's
perceptions and responses that result from the use or
anticipated use of a product, system or service". Following
this definition, user experience includes many aspects of
users’ perceptions and experiences during and after use,
including emotions, beliefs, and physical and psychological
responses. Because of this, user experience has gained
immense interest in HCI, as it highlights the non-utilitarian
aspects of human-technology interactions, and focuses on
user affect, sensation, and meaning as well as the value of
such interactions in everyday life [55]. Although there is
some controversy regarding the extent to which usability and
user experience differ [8], we will refer to the construct of
usability for the rest of the paper.
Despite the success and wide use of the usability construct, a
variety of issues with its use persists (e.g., [56, 57]). First,
usability is often considered to have different components or
dimensions. The ISO 9241 standard, for instance, separates
components of effectiveness, efficiency, and satisfaction;
other definitions include different dimensions [40, 68, 79].
However, these dimensions correlate differently and
sometimes only weakly [27]. And for some dimensions,
many measures show unclear correlations [43].
Second, usability can be measured both objectively and
subjectively. Examples of objective methods include
usability testing and psychophysiological measurements
(e.g., using functional near infrared spectroscopy (fNIRS) in
usability testing [41, 42, 60] and using
electroencephalography (EEG) to evaluate visualization
effectiveness [6] or to compare two user interfaces for
managing personal photos and storytelling [85]). Examples
of subjective methods include AttrakDiff2 [37] and the User
Experience Questionnaire [54]. All are based on self-reports,
collected verbally or non-verbally, and in-situ or post tasks.
The relationship between subjective and objective measures
is mixed, but some findings suggest much lower correlations
than would be expected [27]. Even self-reports on usability
may differ. Lindgaard and Dudek [57] found differences
between ratings and interviews, suggesting that “rating scales
and interview statements may tap different interface
qualities.”
Third, the specific relationship between usability and
aesthetic appeal has been widely researched (e.g., [7, 57, 58,
82]). One key onus of this debate concerns whether “what is
beautiful is usable” [57, 82], that is, whether and how
immediate impressions of aesthetics influence subsequent
performance and assessment of usability.
In sum, usability is widely used and beneficial to HCI.
However, as a construct it remains evasive, with several open
issues regarding exactly what usability is and how it
compares to related concepts (e.g., aesthetics). Here, we limit
the scope of user experience to perceived usability to help
address in particular the first and third question above about
the usability construct.
Functional magnetic resonance imaging (fMRI)
fMRI is a functional neuroimaging procedure that uses MRI
technology to measure brain activity by detecting associated
changes in blood flow [44]. The most common approach to
fMRI uses the Blood Oxygenation Level Dependent (BOLD)
contrast [44]. BOLD fMRI allows us to measure the ratio of
oxy- (oHb) to deoxy-haemoglobin (dHb) in the blood. This
does not directly measure neuronal activity. However, it does
measure the metabolic demands (or oxygen consumption) of
active neurons. Images constructed from fMRI during
performance of different tasks reflect the parts of the brain
that are active, and may reveal the brain structures that are
activated together. This technique has higher spatial
resolution and more accurate activation localization
compared to EEG and fNIRS.
The human cerebral cortex is mapped by divisions into
different functional areas known as Brodmann’s areas. A
Brodmann Area (BA) is a region of the cerebral cortex in the
human brain, defined by its histological structure and
organization of cells. Each BA can correlate with more than
one cortical function and vice versa. For example, BA 1/ 2/ 3
are the primary somatosensory cortex which contains tactile
representation of our body ([52]). In contrast, BA 10
occupies parts of superior frontal gyrus and the middle
frontal gyrus. Therefore, reporting neural activations using
fMRI should include information of both systems (cortical
function area and BA number).
Functional MRI has been used in HCI to help understand
cognitive functions during interactive tasks. Pine et al. [73],
for example, analyzed the brain areas activated during the
performance of a navigation task. Baumgartner et al. [9]
studied brain activations associated with the experience of
presence during a video of a virtual experience. A guide on
conducting fMRIs studies can be found in [23].
Recent neuroimaging studies using fMRI have shown that
user experience parameters might be reflected in brain
activity measurements. In these studies, activated brain
regions were investigated and associated with certain
parameters. However, a common problem with interpreting
fMRI is reverse inference, in which certain brain regions are
alleged to indicate the activation of previously-labeled
cognitive processes [23]. This problem arises because a
particular brain region is seldom associated with only a single
cognitive process [75]. Nonetheless, reverse inference is still
useful in the discovery of interesting new facts about the
mechanisms underlying a cognitive task. In fact,
philosophers have previously argued that reverse inference
(or ‘abductive inference’ in [71]), is an essential tool for
scientific discovery [75, 76]. We mention this because the
later has to carefully guard against mistakes in inferences
about brain areas related to perceived usability.
Functional MRI studies of Usability and Aesthetics
Due to its advantage of opening the “black box” underlying
user experience, fMRI has been employed in several studies
in HCI, especially for exploring virtual reality (VR). For
example, Sjölie et al. [80] used fMRI to investigate the
influence of two VR parameters, 3D-motion and interactivity
for a mental rotation task. They found the most significant
activations during the mental rotation task to be in the
superior parietal lobe and occipital lobe. Clemente et al. [21]
employed fMRI to investigate the sense of presence during a
VR-free navigation task and found that frontal, parietal, and
occipital regions were activated during the free virtual
navigation. Anderson et al. [5] used fMRI to study users’
habituation to security warnings. Their findings showed that
the regions involved attentional processing (left and right
superior parietal cortex) had higher activation for
polymorphic than for static warnings, while the opposite was
true in regions related to memory retrieval processes
(bilateral medial prefrontal cortex and the left retrosplenial
cortex). However, we are not aware of any study that has
investigated the construct of usability or perceived usability
using fMRI. Because of this, any first step of identifying the
neural correlates of usability is exploratory, with no strong a
priori hypothesis, rather than theory driven. This is similar to
how fMRI studies on aesthetics started (e.g., [50]).
In contrast to usability, aesthetics has been investigated
widely using fMRI to study the neural basis of the perception
and experience of aesthetics. It has been shown that the
pleasure elicited in people when they look at beautiful object
is linked to general reward circuitry [51]. For example, when
subjects passively viewed different types of faces, a
significant effect was seen in the nucleus accumbens (NAc)
[2, 50], an area close to the prefrontal cortex, and this effect
was particularly seen in response to the more attractive
female faces. In addition, activation of the caudate part of the
striatum, which has an established role in processing reward
related information [25], has been found to be correlated with
the attractiveness of faces that held eye-contact with the
subjects [47], and also with positive words [35]. This area,
along with the cingulate gyrus and the middle frontal gyrus
have been reported to be activated when judging the
aesthetics of paintings [49, 83], pictures of soft drinks [69],
and different types of faces [2]. A more detailed review can
be found in [51] and [20].
In addition, as there might be a high correlation between the
construct of usability and aesthetics, a topic that is currently
being debated in HCI, an fMRI study will be helpful in
answering this question. This can be done by comparing and
contrasting the brain areas (and their cognitive functions)
activated when looking at the perceived usability and
aesthetics of the same type of visual stimuli.
EXPERIMENT 1: BRAIN RESPONSES TO WEBPAGES
The goal of Experiment 1 is to investigate whether
differences in stimuli with respect to perceived aesthetics and
perceived usability could be detected through fMRI scanning.
We also wanted to investigate whether brain areas associated
with perceived usability and with perceived aesthetics
overlap. We looked at perceived usability and perceived
aesthetics because (a) those are important to the discussion of
the construct of usability, were prominent in reviews of
usability measures (e.g., [8]), and were used in earlier studies
of webpages (e.g., [59]) and (b) because they are feasible to
study in an fMRI scanner, which has the benefit of evaluating
users’ perceptions (of which perceived usability is one
element).
Design
Independent Variables and Design
The study looked at two factors, perceived usability and
perceived aesthetics, with unique stimuli for each. The
factors were manipulated through websites that scored high,
medium, or low on these measures in a pre-study. Thus, the
independent variables are perceived usability and perceived
aesthetics, each with three levels (low, medium, and high).
Selection of stimuli
We collected 400 webpages as candidates for the stimuli to
be used for the main study. The aim was to find web pages
that “appear particularly usable, beautiful, stimulating or
useful, or conversely, particularly unusable, ugly, dull, or
irrelevant.” All pages were in English and were captured as
screenshots of 1920x1080 resolution. This procedure,
including the rating described below, was similar to other
studies on the visual perception of webpages [59, 67, 82].
On the crowdsourcing intermediary site CrowdFlower [22],
492 persons rated screenshots of webpages. Each rated five
to fifteen pages on a variety of measures, including:
Perceived usability, measured by a single question “How
do you rate the usability of this web page?” (on a seven-
point scale from “unusable” to “usable” [82]).
Perceived aesthetics, measured by a single question,
“How do you rate the beauty of the webpage?” (on a
seven-point scale from “ugly” to “beautiful” as in [36]).
Participants also rated the webpages on their hedonic and
pragmatic qualities (with eight questions from a shortened
version of AttrakDiff2 [37]), goodness (as measured by a
seven-point differential from “bad” to “good”, as in [59]),
and perceived aesthetics (measured by six questions from
[81]. The questions covered so-called classical aesthetics,
e.g., “clear”, and expressive aesthetics, e.g., “sophisticated”).
We did not use the last question (expressive aesthetics)
except to check our selection of stimuli, as this would have
required a further experimental construct. Participants could
not interact with the webpages.
To select the stimuli to be used for the fMRI study, we
ranked the webpages on the dimensions of perceived
usability and perceived aesthetics separately to form three
groups of the top 20% (80 webpages), middle 20%, and
bottom 20% of each dimension. To ensure the least
interaction between measures, we selected 25 webpages from
each group that had the greatest difference in rank between
perceived usability and perceived aesthetics, for a total of
150 unique webpages (Table 1). Here the correlations
between perceived beauty and usability were similar, as
found in previous studies (e.g., Tractinsky et al. [81], Tuch et
al. [82]). Thus, the two attributes seem to be consistently
related. In this design we use different pages for ratings of
beauty and usability so that recognizing a stimuli would not
generate spurious brain activations.
Beauty (N=75) Usability (N= 75)
Low Med High Low Med High
Perceived
Usability
4.63
(.51)
4.99
(.40)
5.37
(.36)
3.89
(.52)
5.24
(.14)
6.08
(.23)
Perceived
Aesthetics
3.20
(.47)
4.89
(.12)
5.98
(.17)
4.37
(.52)
4.04
(.55)
5.07
(.51)
Table 1. Average ratings (with standard deviation) of webpages
selected for the fMRI study.
Functional MRI scanning method
Participants
Eight right-handed participants (4 males; mean age 27.4 ±
6.0) participated in the experiment. The participants were
screened using a clinical questionnaire to ensure that none
had a current or prior history of head injury, learning
disability, or psychiatric illness. All participants were
reported to be free of psychotropic medication and had
normal or corrected-to-normal vision. Each participant gave
written informed consent after the explanation of the
experimental protocol, as approved by the local Ethics
committee. The participants were paid $15.
Procedure
Perceived usability and perceived aesthetics were separated
in the experimental design and treated as two separate
investigated factors (see Figure 3). For each factor, webpages
of three levels were shown randomly to participants. This
followed previous studies (e.g., [46]) and was also done to
reduce the boredom of participants if webpages of the same
category are shown continuously.
As a consequence, participants viewed a total of 150
webpages representing two factors (perceived aesthetics and
perceived usability). Each factor had three levels (Low,
Neutral, and High), and each level had a group of 25
webpages. Webpages for all levels of a factor (75 in total)
were presented in a random order between levels. The order
of presentation of factors (one followed by the other) was
counter-balanced between participants. The participants were
instructed before viewing the webpages for each factor that
they would be rating either aesthetics or usability. The
instruction was provided on a screen with only the word
BEAUTY or USABILITY at the screen’s center for 2s. All
webpages were converted into 1024 x 768 resolution to
match the resolution of the fMRI display.
Figure 2. Setup of a participant just before entering the fMRI
machine. The participant had two keypads with two keys each,
and a head mounted mirror to see the screen at the back.
Figure 3. Procedure for one experiment, where the beauty (i.e.,
perceived aesthetics) was viewed and rated first.
Figure 4. Procedure for one trial
Each trial started with a fixation cross displayed at the middle
of a blank screen for a random duration (Inter-Stimulation
Interval - ISI) between 8 and 12s. This randomized ISI was
designed to reconstruct the BOLD response with better
precision. Next, a webpage was shown for 5s, which gave the
participant enough time to observe and form an opinion. A
blank screen with a cross at the middle was then displayed
for 3s, after which the participants were asked one of the two
questions corresponding to the factor being tested:
Rate the usability of the last shown picture
Low Neutral High
or
Rate the beauty of the last shown picture
Low Neutral High
The participants were given unlimited time to answer the
question for each webpage. There was a break of 30s
between the ratings of perceived aesthetics (beauty) and
BEAUTY USABILITY
2s View & rate “beauty”
2s View & rate “usability”
75 webpages 75 webpages30sBREAK
8-12s 5s 3s
RATING
perceived usability (usability), see Figure 3. Figure 4 shows
the timing of each trial. Participants were instructed to rate
the perceived aesthetics of a webpage by its overall visual
attractiveness and appeal, and to rate perceived usability of a
webpage by its ease of use and navigation, and ease by which
information could be obtained. The stimuli exposure time
(5s) follows other works that have presented visual stimuli in
fMRI [5, 30, 45]. Jacobs et al. [45] argued that longer
presentation time of visual stimuli lead to activation patterns
associated with a deeper processing of the stimuli. However,
we had to balance the duration of the scanning (1 hour) and
the number of stimuli to be presented.
After being in the MRI scanner, participants went through all
the webpages in the stimuli set again and inputted their
detailed ratings, using the same questionnaires as in the
CrowdFlower pre-study. This was done because the
participants who performed the fMRI scanning were different
from those who completed the pre-study, and allowed us to
check if answers were consistent between the two groups.
The questionnaire was completed on a laptop, outside the
MRI scanning room. Participants entered their answers on a
7-point Likert scale, as in Figure 5.
Figure 5. Post-scan webpage rating task
The experiment lasted about 1h 15m per participant, with
60m scanning and 15m for the post-scanning questionnaire.
MRI Data Acquisition
Experiments were run with the Psychophysics Toolbox [14].
Neuroimaging data was acquired with a 3T Siemens
Magnetom Skyra MRI scanner. Subjects were placed in a 32-
channel RF head coil with a mirror mounted on the coil to
view the webpages. Soft padding was placed on either side
of the head to limit head movement during fMRI image
acquisition. Webpages were presented on a screen mounted
at the back of the scanner, which participants could observe
through the attached head-mounted device. Ratings of
webpages were collected using a 4-key fiber-optic Lumina
response box (see Figure 2). Functional images were
acquired with the following parameters for T2*-weighted
gradient echo sequence: 64 x 64 matrix, TR = 2,500ms, TE =
30ms, FOV = 192mm, flip angle 90. Thirty-six slices were
acquired to cover the whole brain with an interleaved slice
acquisition and 3 x 3 x 3 mm voxel resolution. In addition, a
high resolution structural scan was acquired with T1-
weighted MP-RAGE sequence (256 x 256 matrix, 192 slices
in sagittal plane and a 0.9 x 0.9 x 0.9 mm voxel).
Analysis Method
Statistical Parametric Mapping software (SPM8; Welcome
Trust Centre for Neuroimaging) was used for image data
processing and analysis. The first seven scans were excluded
from the analysis to eliminate the decay of the fMRI signal
associated with the moment when magnetization reaches
equilibrium. Functional data from each run were aligned to
the run nearest in time to the acquisition of the structural
scan. Then, the mean image produced during the process of
realignment, and the realigned images were co-registered to
the high-resolution T1 anatomical image. All images were
spatially normalized to standard MNI space. They were then
spatially smoothed using 10-mm FWHM Gaussian kernel to
facilitate group analysis, and a high-pass filter of 1/128 Hz
was used to eliminate low-frequency components. The
functional imaging data were modeled using a boxcar
function with head motion parameters as unrelated
regressors. Parameter estimates for each condition (three
types of stimuli) were calculated from a general linear model
(GLM) based on the hemodynamic response function with
overall grand mean scaling. Whole-brain statistical
parametric mapping analyses were performed. The t-contrast
images were generated for comparison at each voxel.
Statistical tests were first assessed in individual subjects, and
then random effect analyses were conducted based on
statistical parameter maps from each individual subject, to
allow population inference. A one-sample t-test was applied
to determine group-level activation for intelligibility effect.
This process was similar to previous works [26, 29].
RESULTS OF EXPERIMENT 1
Behavioral results
On average, it took participants 0.93s (SD = 0.49s) to rate a
webpage in the scanner. A t-test shows no significant
difference in response time between perceived aesthetics (M
= 0.90s, SD = 0.71s) and usability (M = 0.96s, SD = 0.79s), p
= 0.22. An ANOVA with Bonferroni correction shows no
significant difference in response time within the perceived
aesthetics groups (p = 0.67). However, participants spent less
time rating high usability webpages compared to low
usability ones (p < 0.005).
We compared the ratings obtained from CrowdFlower (called
cloud ratings), from participants while inside the scanner
(called inscan ratings), and from participants in the post-scan
task (called postscan-ratings). A Pearson’s correlation test
between cloud-, inscan-, and postscan ratings shows that
participants were fairly consistent in rating webpages during
and after the scans (r = 0.74). However, it also shows that
judgment regarding the perceived usability of a webpage is
fairly personal, with cloud- and inscan ratings having r =
0.52. Figure 6 illustrates these different types of responding
time.
Figure 6. Inscan responding time with standard error bars.
Testing of Brain Activity
To analyze the fMRI data, we contrasted conditions of
interest corresponding to each level of perceived aesthetics
and usability by assigning values of 1 and -1 to the regressors
of interest, and 0 to all other regressors. The brain activity
occurring with when there were no stimuli presented was
used as baseline condition and was assigned a value of 0 in
all contrasts. Following the experimental design, we analyzed
perceived aesthetics and usability separately.
Specifically, we first did pair-wise comparisons between
baseline and stimuli for aesthetics and usability. We then did
pair-wise contrasts of brain regions that were active during
the different levels of usability and aesthetics (e.g., low vs.
high). That helped to identify regions involved in different
levels of usability and aesthetics.
Brain Correlates of Perceived Aesthetics
We observed a large number of activation areas across the
brain during the judgment of perceived aesthetics. Table 2
shows the details of these areas for the contrasts of perceived
aesthetics vs. baseline. We found several regions in frontal
lobe, temporal lobe, limbic lobe, and basal ganglia that were
activated in the contrast of perceived aesthetics vs. baseline
(p < 0.001). Table 2 also shows the details of activation areas
for the comparisons of different levels of perceived aesthetics
(low/medium/high). We found activations in the frontal lobe,
parietal lobe, and the limbic lobe for both the high vs. low
and high vs. medium comparisons of perceived aesthetics.
Discussion of Perceived Aesthetics
The brain areas found to be specific to perceived aesthetics
are in line with those found for non-webpage stimuli. For
instance, we observed strong activation in the left middle
frontal gyrus, suggesting that people rate the perceived
aesthetics of webpages similarly to pictures and architecture
stimuli [46, 84]. This may be because most webpages include
commodity design elements such as columns, frames and
headings. These commodity design elements add
architectural features to the static webpage, thus capturing
users’ attention. In addition, the activations seen in the
bilateral inferior frontal gyrus, claustrum, and insula are
similar to a previous study demonstrating baby schema [30],
suggesting that the participants may perceive the features of
webpages as cute or motivating a sense of caretaking in the
subjects. Activation of the sub-gyral, an area associated with
the hippocampus [18], suggests that the perceived aesthetics
of webpages are also judged based on visual aesthetic
perception. Thus, our study found activation areas related to
perceived aesthetics that were also reported in previous
studies involving aesthetic stimuli.
Structure BA k x y z p
Aesthetics - baseline
Frontal lobe
Middle Frontal Gyrus L 10 168 -40 42 14 1.3E-5
Inferior Frontal Gyrus R 45/47 68 49 20 11 0.0001
Medial Frontal Gyrus R 9 27 5 44 18 0.0002
Medial Frontal Gyrus L 10 27 -1 47 12 0.0004
Temporal lobe
Superior Temporal Gyrus L 22 41 -45 -5 -4 1.6E-5
Limbic lobe
Anterior Cingulate R 24 260 1.86 31 5.44 9.9E-6
Insula R 13 103 43 -6 3 3.5E-5
Insula L 13 151 -43 -14 3 4.5E-5
Sub-Gyral (Hippocampus) R 28 35 -45 4 3.9E-5
Cingulate Gyrus L 23 25 1.4 -27 24 0.0002
Cingulate Gyrus R 31 25 1.29 -31 32 6.9E-5
Basal ganglia
Lentiform Nucleus /
Putamen L 151 -29 -11 -2 0.0005
Claustrum L 110 -26 -13 22 7.8E-6
Caudate (Caudate Tail) R 28 29 -34 8 0.0007
Aesthetics (High – Low)
Frontal lobe
Middle Frontal Gyrus L 46 12 -46 19 20 2.5E-5
Parietal lobe
Angular Gyrus L 39 6 -29 -62 31 1.8E-5
Precuneus L 19 6 -32 -68 39 0.0007
Limbic lobe
Cingulate Gyrus R 24 5 4.12 5.23 38.17 4.4E-5
Aesthetics (High – Med)
Frontal lobe
Medial Frontal Gyrus R 6 50 1 1 57 0.0002
Medial Frontal Gyrus L 6 8 -7 0 59 0.0002
Paracentral Lobule R 31 10 4 -13 47 0.0003
Paracentral Lobule L 31 7 -2 -21 46 1.3E-5
Parietal lobe
Inferior Parietal Lobule L 40 7 -57 -27 26 0.0002
Inferior Parietal Lobule R 40 7 57 -29 36 4.0E-5
Limbic lobe
Cingulate Gyrus R 24 11 1.31 4.98 40.81 0.0002
Cingulate Gyrus L 24 10 -7.1 -14.8 41.49 0.0002
Table 2. Regions activated in the parametric analyses of inscan
ratings for perceived aesthetics. Here, L/R refers to the Left/
Right cerebrum of the brain; BA refers to the Brodmann area
number; k is the cluster size of the activation, measured in the
number of voxels; x, y, z are coordinates in Talairach space; and
p is the probability of the comparison.
Brain Correlates of Perceived Usability
Similar to the findings for perceived aesthetics, the
comparison of Perceived Usability vs. baseline revealed
activations across the brain in the frontal, temporal, and
limbic lobes (p < 0.001). The activated areas are shown in
Figure 1 and Table 3, and are discussed further in the next
paragraph. Specifically, we found that in some areas in the
frontal, parietal, and limbic lobes, and in the basal ganglia,
responded to the differences in perceiving different levels
(high, medium, and low) of usability. Here, only the high vs.
low and medium vs. low comparisons of perceived usability
yielded activations (p < 0.001), while no significant
activation was found for high vs. medium.
0
0.5
1
Low Med High Low Med High
Beauty Usability
Discussion of Perceived Usability
Perceived usability was found to be associated with increased
activation of several premotor areas, including the superior
frontal gyrus (BA6) and the precentral gyrus (BA6). The
latter showed increased activation with increased usability.
One interpretation of BA6 is that it is associated with
planning complex movements, as it is the site of the premotor
cortex and supplementary motor cortex [19]. Similarly, the
right superior frontal gyrus (BA10) is involved in planning to
use something [19]. In addition, the right superior frontal
gyrus (BA10) was found to be activated in a previous study
of reward in an N-back task (where subjects watch a
sequence of stimuli and need to recall whether the current
stimulus was the same as the previous Nth-stimulus) [74]. The
region was also found to be activated in a self-aware state
[31]. This suggests that the process of planning to use a
webpage and self-reflection on how to use the webpage
viewed were stimulated. The activation of these areas was
specific to usability.
Structure BA k x y z p
Usability - baseline
Frontal lobe
Middle Frontal Gyrus L 9 30 -32 26 29 7.3E-5
Medial Frontal Gyrus L 9/10 26 -4 56 10 0.0005
Medial Frontal Gyrus R 8 16 1 28 38 2.1E-5
Superior Frontal Gyrus L 8 24 -2 35 54 1.8E-5
Superior Frontal Gyrus R 6 12 4 15 55 0.0001
Temporal lobe
Middle Temporal Gyrus R 19/37 22 35 -57 14 0.0001
Limbic lobe
Cingulate Gyrus (31) L 31 62 -21 -45 30 4.6E-6
Posterior Cingulate (31) R 31 20 1.29 -58 24 0.0001
Usability (High – Low)
Frontal lobe
Superior Frontal Gyrus R 10 17 27 49 26 0.0001
Fusiform Gyrus L 37 11 -46 -60 -12 2.3E-5
Cingulate Gyrus R 24 11 4 2 41 3.8E-5
Precentral Gyrus R 6 5 49 -8 32 0.0004
Limbic lobe
Cingulate Gyrus L 32 7 -1 23 34 0.0007
Cingulate Gyrus R 32 7 7 23 34 7.4E-5
Anterior Cingulate R 32 7 7 26 27 0.0001
Parietal lobe
Inferior Parietal Lobule L 40 9 -49 -32 47 0.0001
Postcentral Gyrus L 2 9 -52 -23 45 0.0006
Basal ganglia
Lentiform Nucleus / L 8 -24 10 -8 4.6E-5
Claustrum L 8 -23 22 10 0.0001
Usability (Med – Low)
Parietal lobe
Inferior Parietal Lobule L 40 5 -41 -45 38 1.4E-5
Table 3. Regions activated in the parametric analysis of inscan
ratings for perceived usability.
Usability was also related to activation of the medial and
bilateral superior frontal gyri (BA8). Activation of these
areas was specific to perceived usability, and they are often
related to linguistic processing and reading [66]. In some
studies, they have also been implicated in higher-order
expectancy and utility [17]. This suggests that even if a
webpage is only viewed for 5s, users look at the quality of
the headings in the webpage when perceiving usability.
Increases in usability were also found to be associated with
activation in areas of the somatosensory cortex associated
with touch (postcentral gyrus, [77]). This has been seen in the
process of observing touch actions [12], which is a kind of
“mirror effect” of touch. This effect was observed in our
study, suggesting that nowadays webpages are often viewed
on a touch-based device, which might lead participants to
assess the usability of a webpage by asking themselves:
“What happens if I go there (by touching it)?”
Finally, the left middle frontal gyrus (BA9) was activated
when assessing usability. This region is related to emotion
[23] and was specific to perceived usability. This area has
been shown to be related to the implementation of reappraisal
(the emotional regulation strategy that involves changing the
trajectory of an emotional response) in daily life [32]. This
suggests that the affective influence of a webpage is a key
factor in perceiving usability.
Discussion of Experiment 1
The results from this experiment show an overlap in brain
activation when perceiving aesthetics and usability, in the left
medial frontal gyrus (BA 10) and the inferior parietal lobule
(BA 40). The left medial frontal gyrus was likely activated in
response to commodity design elements when subjects were
asked to judge the usability of a static webpage. This effect
has also been seen in the parametrical effects of aesthetics
and familiarity ratings [13] and aesthetic judgment of
pictures [46]. Also, the activation of the inferior parietal
lobule has been observed in response to infant faces versus
crosshair baby schema levels. The overlap in these regions
between aesthetics and usability may be interpreted in several
ways. First, it may be that the assessment of some
components of perceived usability is replaced by the
(presumably simpler) assessment of perceived aesthetics.
Second, perceived aesthetics may form part of perceived
usability, causing the activation areas to overlap.
The results described above show brain activation areas
similar to those found in previous studies. This suggests that
the participants perceived the webpages in a similar way to
non-webpage stimuli. However, activation of these areas has
been observed in a combination of several studies of different
types of stimuli (paintings, landscapes, faces, etc.). This
suggests that the participants perceived the aesthetics of
webpages as a combination of separate components,
according to the different components of the webpage itself.
Similarly, because of the overlap between perceived
aesthetics and usability, it can be suggested that participants
also perceived usability of webpages as a combination of the
different components’ aesthetics and usable values of the
being-viewed webpages.
EXPERIMENT 2: DYNAMIC STIMULI
In Experiment 1, we found that viewing beautiful webpages
triggers similar activation areas in the brain as have been
found in previous studies. These same areas were also
activated when participants viewed webpages with low
usability. Many models of usability acknowledge that initial
exposure to an interface differs from after interacting with it
(e.g., [3, 48]). For example, measures of aesthetics and of
usability are often strongly correlated at the time of initial
exposure to an interface, but less correlated – if at all – after
interaction with an interface. Thus, while the first study
presented initial data on the associations between activities in
brain networks and usability and aesthetic ratings, these
relationships might differ in interactive systems.
To investigate this further, we added dynamics to the stimuli
and performed the same experimental protocol. This was
done to check whether people evaluate perceived aesthetics
and usability in an interactive user interface the same as with
static webpages (where there is no interactivity between the
application and the user). Interactive here refers to interfaces
that change appearance, as demonstrated by videos showing
the user interface changing, either through a fictive user
interacting with it (indicated through the mouse moving) or
to show a notification. This approximates real interaction,
and earlier work on user interfaces has used this approach
([70]). Furthermore, previous works in neuroscience suggests
that at least some reactions to observing people use tools are
similar to the reactions people have when using the tool
themselves (e.g., [64, 78]).
Design & Method
Selection of Stimuli
From Experiment 1’s results, we saw that the BOLD signal
response for the medium group was not significant different
from the activation of the high group (usability) or low group
(aesthetics) and with limited contrast between them.
Consequently, in this experiment, we created videos of
simple webpage interactions based on 100 webpages from
the previous experiment (25 from each group: low aesthetics,
high aesthetics, low usability, high usability). In addition, we
added 10 additional videos to each group that clearly showed
low/high aesthetics or usability. This was done to increase
the activation-based contrast by having a larger number of
samples for each group.
Each video contained a normal webpage interaction such as
the mouse scrolling down, or a mouse click to change page
(1-2 times). The mouse cursor and its trajectories were
visible in the video. All animations were captured using
Camtasia Studio Screen Recorder and saved in MP4 and
1080p video format. They were later converted to 1280x720
size (720p) to fit the MRI projector resolution. On average,
the videos had a length of 7.56s (± 1.19s).
fMRI Procedure
We employed the same procedure for video stimuli as with
the previous experiment on static webpages (Figure 3 and
Figure 4). In short, there was a blank screen with a cross
displayed at the middle for 8-12s, followed by the video.
After that, the same blank screen appeared for 3s, and then a
question popped up to ask the participants for a rating of the
video. Similar to the previous experiment, participants were
instructed to rate the perceived aesthetics of the webpage by
its overall visual attractiveness and appeal, and rate perceived
usability by its ease of use and navigation, and ease of
obtaining information.
Eight participants (4 male), aged from 18 to 32, volunteered
for this experiment. They were all right-handed (by self-
report) and did not participate in the previous experiment.
Similar to the previous experiment, the participants
completed a clinical questionnaire to ensure that none had a
current or prior history of head injury, learning disability, or
psychiatric illness. All participants were free of psychotropic
medications and had normal or corrected-to-normal vision.
They were paid $15 for their time. None of the volunteers
had participated in the previous experiment.
Also similar to Experiment 1, after being scanned in the MRI
scanner, the participants went through all videos in the
stimuli set and inputted their detailed ratings. This was done
to see whether the answers were consistent between the
experimental and pre-test groups. The questionnaire was
completed on a laptop, outside the MRI scanning room.
Participants entered their answers on a more detailed scale
(7-point Likert scale). The total experiment time for each
participant was about 1 hour 15 minutes, with 1 hour for
scanning and 15 minutes for the post-scan questionnaire.
Results
Behavioral
On average, it took participants 1.07s (±1.23s) to respond to a
rating question in the scanner. ANOVA with Bonferroni
correction shows a difference in response time between the
perceived aesthetics and usability groups (p < 0.05).
Participants spent a slightly longer time to judge the
perceived usability of the stimuli (mean 0.996s ± 0.226)
compared to the perceived aesthetics (mean 1.155s ± 0.226).
A post-hoc test reveals no significant difference between
high and low ratings within each group of perceived
aesthetics and usability (p > 0.05).
Brain activity for Perceived Usability and Aesthetics
We used a similar method as in the first study to contrast the
conditions of interest corresponding to each level of
aesthetics and usability by assigning values of 1 and -1 to the
regressors of interest, and 0 to all other regressors. We also
analyzed aesthetics and usability separately.
Table 4 shows the activation areas for the contrasts of
perceived aesthetics vs. baseline and perceived usability vs.
baseline. We did not find significant activation areas (k >= 4)
when comparing different levels of perceived aesthetics and
usability. We observed activations in the contrast of
perceived aesthetics vs. baseline in the frontal lobe (bilateral
medial frontal gyrus, BA6, and the precentral gyrus, BA6).
For perceived usability vs. baseline, there was increased
activity in the precentral gyrus (BA4). Interestingly, all
activations related to aesthetics were located in BA6, which
is the site of the premotor cortex and supplementary motor
cortex. This area is related to early planning of movements
and events, as discussed earlier. In contrast, the activations
caused by usability vs. baseline were found in the precentral
gyrus (BA4), which was not activated for perceived
aesthetics (Figure 7). This is the site of the primary motor
cortex, which controls all voluntary movements. This
suggests that perceived usability is somewhat more related to
action than perceived aesthetics.
Structure BA k x y z p
Aesthetics - baseline
Frontal lobe
Medial Frontal Gyrus R 6 19 15 -16 53 0.0003
Medial Frontal Gyrus L 6 4 -13 -13 55 0.0002
Precentral Gyrus R 6 5 43 -12 35 6.2E-5
Basal ganglia
Caudate Tail L 5 -21 -37 12 0.0002
Usability - baseline
Frontal lobe
Precentral Gyrus R 4 14 23 -22 58 1.4E-5
Table 4. Regions activated in the parametric analyses of inscan
ratings for perceived aesthetics and usability when viewing
animated webpages.
Figure 7. Activations in the precentral gyrus (BA4) for
perceived aesthetics vs. baseline (left) and perceived usability vs.
baseline (right).
Discussion of Experiment 2
The results show that for the contrast of perceived aesthetics
vs. baseline, the bilateral medial frontal gyrus was activated.
Its activation has been found during observation of cartoons
and story comprehension involving animation stimuli [28].
Also, activations in the caudate tail have been shown to be
associated with ‘what’ and ‘where’ the information was
received and guides the rapid eye movement to visual objects
[86]. This area related to eye movements is likely linked to
the recorded interaction embedded in the animated
webpages. This result suggests that participants assessed not
only the perceived aesthetics of the webpage’s but also the
interaction.
OVERALL DISCUSSION
Our results show that there are distinct differences between
the neuroanatomical areas related to perceived usability and
those related to perceived aesthetics. We have shown this
both as differences between a baseline condition and viewing
of webpages, and as differences between seeing webpages
that have been assessed as high, medium, or low on
perceived usability and aesthetics. We have also shown that
recorded interactivity added to webpages attenuates these
differences and allows fewer distinct brain areas to be
discerned. Below, we discuss these results.
The brain components of perceived usability
A key finding of this paper is that perceived usability has a
brain basis that differs from that of perceived aesthetics. We
find this remarkable because (a) exposure time to webpages
was limited, (b) only static images were shown in
Experiment 1, and (c) many studies find moderate
correlations between assessments of perceived usability and
perceived aesthetics (e.g., [82]). Still, there is something
particular to usability that emphasizes its prominent status in
human-computer interaction. We are particularly excited
about four connections in brain areas specific to usability.
First, our results suggest that emotion is integral to perceived
usability. We found activation in the fusiform gyrus, which is
involved in emotional facial expression, and in the middle
frontal gyrus (BA9), which shows activity during emotional
suppression [4]. Another evidence is the activation seen in
the superior frontal gyrus (BA10) and the anterior cingulate
gyrus in the contrast of Usability vs. baseline. Both have
been found to show activations during the self-regulation of
emotion [10]. In contrast, many current conceptions of
usability do not clearly specify a role for emotion in usability
(e.g., ISO 9241); our findings suggest that emotion does
indeed play a part.
Second, the findings of the first experiment suggest that the
anticipation of physical interaction is a crucial part of
perceived usability. During the judgment of usability of static
webpages, the precentral gyrus (BA6) was activated, which
may relate to the planning of movements related to the
webpage. Interestingly, some studies relate this area to the
sense of touch; others to the notion of affordance [24]. In the
second experiment, during the judgment of videos of
webpages, some activation was found in the precentral gyrus
of BA4, which is the site of the primary motor cortex (i.e.,
directly related to voluntary movement). This suggests that
the interactions depicted in the video trigger higher
anticipation of interaction with the webpages.
Third, the area activated specifically for perception of
usability is related to task intention [15, 39, 63]. This
suggests that perceived usability might implicate some notion
of task because assessing usability activates task areas. Some
studies relate this area to affordance [34, 62]. Task intention
is key to most definitions of usability, for example, ISO
9241, which says that usability is relative to the task. Here,
even though many web pages were related to leisure and
scored high in pretesting on hedonic quality [37], these areas
were still active.
Fourth, Experiment 1 revealed activity in several brain areas
related to linguistic processing [61], categorization, and
rational thought – in short, higher-order thinking. This could
be interpreted in the way that recognizing categories, an
essential aspect of navigation, is crucial to assessing usability
and could also be distinct for usability. Even without given
the opportunity to interact, people still attempt to categorize
and recognize navigation options on the webpages.
Usability and aesthetics of webpages
From the first experiment, we found some overlap between
perceived aesthetic and usability in terms of brain activation.
This is in line with previous literature, suggesting that stimuli
with high-perceived usability are also highly aesthetic, and
vice versa. Our results show that the left medial frontal gyrus
(BA10, found in the parametrical effects of aesthetics and
familiarity ratings, and aesthetics judgment of pictures) and
the inferior parietal lobule (found in response to infant faces
versus crosshair baby schema levels) were activated with
both perceived usability and aesthetics. This suggests that
users might assess some components of perceived usability
by substituting an assessment of perceived aesthetics.
Likewise, perceived aesthetics may form part of perceived
usability, leading to the overlap between these areas.
We found neural correlates of perceived usability during the
judgment of static and dynamic stimuli. Although the stimuli
during both studies were webpages, which are complex and
combine multiple elements, we found that most activation
was in areas identified in previous studies of visual aesthetic
stimuli (e.g., aesthetics and familiarity ratings [13], faces
[72], aesthetics of contour objects [84], baby schema [30]).
Our findings confirm that a complex stimulus such as a
webpage has its aesthetics judged in the same way as other
simpler stimuli.
The results indicate that the right medial frontal gyrus (BA9)
is activated during perceived aesthetics judgement, although
this has not been investigated previously in the literature on
aesthetics. Functionally, the right medial frontal gyrus (BA9)
is related to moral judgments (e.g., judging a sentence “right”
or “wrong” [65], or probing using moral dilemmas [33]). The
anterior cingulate cortex (BA24) was shown to be activated
during the process of recognizing error [16]. The activations
found in these regions suggest that besides judging whether a
webpage is beautiful, participants may have been judging
whether this webpage design was right or wrong for the
intended purpose. This might have a connection with the task
intention aspect of usability indicated earlier.
Implications for future study and limitations
As our study represents the first attempt to identify the neural
correlates of usability, we could not form a strong a priori
hypothesis, as is normally done with functional imaging of
the brain [23]. There must be a first foundation upon which
to propose hypotheses for further investigation – this paper
serves that purpose. More in-depth studies of usability can
follow our findings to investigate each aspect of usability
individually, such as the effects of emotion, anticipation of
physical interaction, and task intention.
In addition, the brain activations found in our studies were
not just common brain network activations across tasks.
There are two reasons for this. First, the brain activations
found for perceived aesthetics are in line with the literature
on aesthetics, suggesting that they are not just common brain
networks. Second, we have analyzed the distinct activations
of perceived usability (both as contrast vs. baseline and as
differences between levels of usability). Thus, our key results
show (a) the activation areas that are distinct from perceived
aesthetics and (b) the links between brain area function and
the attributes of perceived usability
One limitation of our study is in the experimental protocol.
The fact that aesthetics and usability are interrelated presents
a challenge; many other applications of fMRI can separate
the independent variables much more clearly. One idea used
in earlier work is to extensively vary the questions used to
ask participants about the webpages. This would have
simplified the analysis but would not have allowed us to
manipulate usability and aesthetics as we did.
We have not attempted to deal with objective usability, for
instance, through task completion times or error rates. Also,
we did not treating data as correlational (e.g., correlating
ratings and brain activity). Those are obvious directions for
future work. However, HCI researchers who employ fMRI
technique should consider the difficulty of supporting user
interaction in fMRI scanners: participants need to remain
static while being scanned. Other limitations include displays
with low resolution and the inability to use metal parts in
input devices when in a scanner.
Uses for Evaluation and New User Experiences
One use of our findings is for usability evaluation. We
imagine a potential for automated evaluation whereby
perceived usability may be assessed over time without asking
people, and whereby brain-based evaluations can be used to
help distinguish variants of a user interface. Possibly, the
most promising use of this would be for assessing highly
dynamic stimuli; another idea would be to test the affordance
of an interface this way. Currently, fMRI equipment is
expensive but portable and low-cost MR scanners are being
developed. In addition, we propose dynamically changing
user interfaces based on fMRI or a similar technique. This
would use adaptive user interfaces that relate to some aspects
of usability we have discussed (e.g., anticipation of physical
interaction, higher-order thinking) and adapts interfaces
based on the scans. Our results could also be used to adapt
interfaces on their perceived aesthetics during interaction.
CONCLUSION
Our paper offers the first exploratory step in investigating
usability using a neurological approach. We have used fMRI
to identify brain areas specific to usability and areas shared
with aesthetics. We have also compared and contrasted
perceived usability and aesthetics with the purpose of
offering an explanation for why usability and aesthetics are
correlated, but are not the same. The dimensions of usability
have also been discussed based on an interpretation of the
function of brain areas specific to usability, indicating
emotion, anticipation of physical interaction, task intentions,
and higher-order thinking as key dimensions.
ACKNOWLEDGEMENT
This work forms part of GHOST, a project funded by the
European Commission's 7th Framework Programme, FET-
Open scheme (Grant #309191). We thank Dr. Jade Ngoc
Thai (CRICBristol, University of Bristol) for the help in
providing access to the fMRI scanning facility.
REFERENCES
1. Ergonomics of human system interaction - Part 210:
Human-centred design for interactive systems, in ISO
DIS 9241-210. 2008, International Organization for
Standardization (ISO).
2. Aharon, I., N. Etcoff, D. Ariely, C.F. Chabris, E.
O'Connor, and H.C. Breiter, Beautiful faces have
variable reward value: fMRI and behavioral evidence.
Neuron, 2001. 32(3): p. 537-51.
3. Al-Shamaileh, O. and A. Sutcliffe, The effect of website
interactivity and repeated exposure on user experience,
in Proceedings of the 4th Mexican Conference on
Human-Computer Interaction. 2012, ACM: Mexico
City, Mexico. p. 1-8.
4. Amting, J.M., S.G. Greening, and D.G.V. Mitchell,
Multiple Mechanisms of Consciousness: The Neural
Correlates of Emotional Awareness. The Journal of
Neuroscience, 2010. 30(30): p. 9.
5. Anderson, B.B., C.B. Kirwan, J.L. Jenkins, D. Eargle, S.
Howard, and A. Vance, How Polymorphic Warnings
Reduce Habituation in the Brain: Insights from an fMRI
Study, in CHI. 2015, ACM: Seoul, Republic of Korea. p.
2883-2892.
6. Anderson, E.W., K.C. Potter, L.E. Matzen, J.F.
Shepherd, G.A. Preston, and C.T. Silva, A user study of
visualization effectiveness using EEG and cognitive
load, in EuroVis. 2011, Eurographics Association:
Bergen, Norway. p. 791-800.
7. Angeli, A.D., A. Sutcliffe, and J. Hartmann, Interaction,
usability and aesthetics: what influences users'
preferences?, in Proceedings of the 6th conference on
Designing Interactive systems. 2006, ACM: University
Park, PA, USA. p. 271-280.
8. Bargas-Avila, J.A. and K. Hornbæk, Old wine in new
bottles or novel challenges: a critical analysis of
empirical studies of user experience, in CHI. 2011,
ACM: Vancouver, BC, Canada. p. 2689-2698.
9. Baumgartner, T., D. Speck, D. Wettstein, O. Masnari, G.
Beeli, and L. Jancke, Feeling present in arousing virtual
reality worlds: prefrontal brain regions differentially
orchestrate presence experience in adults and children.
Front Hum Neurosci, 2008. 2: p. 8.
10. Beauregard, M., J. Levesque, and P. Bourgouin, Neural
correlates of conscious self-regulation of emotion. J
Neurosci, 2001. 21(18): p. Rc165.
11. Bias, R.G. and D.J. Mayhew, Cost-Justifying Usability:
An Update for the Internet Age. 2005: Morgan
Kaufmann Publishers Inc.
12. Blakemore, S.-J., D. Bristow, G. Bird, C. Frith, and J.
Ward, Somatosensory activations during the observation
of touch and a case of vision–touch synaesthesia. Brain,
2005. 128(7): p. 1571-1583.
13. Bohrn, I.C., U. Altmann, O. Lubrich, W. Menninghaus,
and A.M. Jacobs, When we like what we know--a
parametric fMRI analysis of beauty and familiarity.
Brain Lang, 2013. 124(1): p. 1-8.
14. Brainard, D.H., The Psychophysics Toolbox. Spat Vis,
1997. 10(4): p. 433-6.
15. Bush, G., B.A. Vogt, J. Holmes, A.M. Dale, D. Greve,
M.A. Jenike, and B.R. Rosen, Dorsal anterior cingulate
cortex: A role in reward-based decision making.
Proceedings of the National Academy of Sciences, 2002.
99(1): p. 523-528.
16. Carter, C.S., T.S. Braver, D.M. Barch, M.M. Botvinick,
D. Noll, and J.D. Cohen, Anterior cingulate cortex, error
detection, and the online monitoring of performance.
Science, 1998. 280(5364): p. 747-9.
17. Casey, B.J., K.M. Thomas, T.F. Welsh, R.D. Badgaiyan,
C.H. Eccard, J.R. Jennings, and E.A. Crone,
Dissociation of response conflict, attentional selection,
and expectancy with functional magnetic resonance
imaging. Proc Natl Acad Sci U S A, 2000. 97(15): p.
8728-33.
18. Cela-Conde, C.J., G. Marty, F. Maestú, T. Ortiz, E.
Munar, A. Fernández, M. Roca, J. Rosselló, and F.
Quesney, Activation of the prefrontal cortex in the
human visual aesthetic perception. PNAS, 2004.
101(16): p. 6321-6325.
19. Chouinard, P.A. and T. Paus, The primary motor and
premotor areas of the human cerebral cortex.
Neuroscientist, 2006. 12(2): p. 143-52.
20. Cinzia, D.D. and G. Vittorio, Neuroaesthetics: a review.
Curr Opin Neurobiol, 2009. 19(6): p. 682-7.
21. Clemente, M., B. Rey, A. Rodríguez-Pujadas, A. Barros-
Loscertales, R.M. Baños, C. Botella, M. Alcañiz, and C.
Ávila, An fMRI Study to Analyze Neural Correlates of
Presence during Virtual Reality Experiences. Interacting
with Computers, 2013.
22. CrowdFlower: AI for your business.
https://www.crowdflower.com/
23. Dimoka, A., How to conduct a functional magnetic
resonance (fMRI) study in social science research. MIS
Q., 2012. 36(3): p. 811-840.
24. Ebisch, S.J., M.G. Perrucci, A. Ferretti, C. Del Gratta,
G.L. Romani, and V. Gallese, The sense of touch:
embodied simulation in a visuotactile mirroring
mechanism for observed animate or inanimate touch. J
Cogn Neurosci, 2008. 20(9): p. 1611-23.
25. Elliott, R., K.J. Friston, and R.J. Dolan, Dissociable
neural responses in human reward systems. J Neurosci,
2000. 20(16): p. 6159-65.
26. Foley, E., G. Rippon, N.J. Thai, O. Longe, and C.
Senior, Dynamic Facial Expressions Evoke Distinct
Activation in the Face Perception Network: A
Connectivity Analysis Study. Journal of Cognitive
Neuroscience, 2011. 24(2): p. 507-520.
27. Frøkjær, E., M. Hertzum, and K. Hornbæk, Measuring
usability: are effectiveness, efficiency, and satisfaction
really correlated?, in CHI. 2000, ACM: The Hague, The
Netherlands. p. 345-352.
28. Gallagher, H.L., F. Happé, N. Brunswick, P.C. Fletcher,
U. Frith, and C.D. Frith, Reading the mind in cartoons
and stories: an fMRI study of ‘theory of mind’ in verbal
and nonverbal tasks. Neuropsychologia, 2000. 38(1): p.
11-21.
29. Ge, J., G. Peng, B. Lyu, Y. Wang, Y. Zhuo, Z. Niu, L.H.
Tan, A.P. Leff, and J.-H. Gao, Cross-language
differences in the brain network subserving intelligible
speech. PNAS, 2015. 112(10): p. 2972-2977.
30. Glocker, M.L., D.D. Langleben, K. Ruparel, J.W.
Loughead, J.N. Valdez, M.D. Griffin, N. Sachser, and
R.C. Gur, Baby schema modulates the brain reward
system in nulliparous women. PNAS, 2009. 106(22): p.
9115-9119.
31. Goldberg, I.I., M. Harel, and R. Malach, When the Brain
Loses Its Self: Prefrontal Inactivation during
Sensorimotor Processing. Neuron, 2006. 50(2): p. 329-
339.
32. Grecucci, A., C. Giorgetta, N. Bonini, and A.G. Sanfey,
Reappraising social emotions: the role of inferior frontal
gyrus, temporo-parietal junction and insula in
interpersonal emotion regulation. Frontiers in Human
Neuroscience, 2013. 7: p. 523.
33. Greene, J.D., R.B. Sommerville, L.E. Nystrom, J.M.
Darley, and J.D. Cohen, An fMRI Investigation of
Emotional Engagement in Moral Judgment. Science,
2001. 293(5537): p. 2105-2108.
34. Grezes, J., M. Tucker, J. Armony, R. Ellis, and R.E.
Passingham, Objects automatically potentiate action: an
fMRI study of implicit processing. European Journal of
Neuroscience, 2003: p. 6.
35. Hamann, S. and H. Mao, Positive and negative
emotional verbal stimuli elicit activity in the left
amygdala. Neuroreport, 2002. 13(1): p. 15-9.
36. Hassenzahl, M., The interplay of beauty, goodness, and
usability in interactive products. Hum.-Comput.
Interact., 2008. 19(4): p. 319-349.
37. Hassenzahl, M., M. Burmester, and F. Koller,
AttrakDiff: Ein Fragebogen zur Messung
wahrgenommener hedonischer und pragmatischer
Qualität, in Mensch & Computer 2003, G. Szwillus and
J. Ziegler, Editors. 2003, Vieweg+Teubner Verlag. p.
187-196.
38. Hassenzahl, M. and N. Tractinsky, User experience-a
research agenda. Behaviour & Information Technology,
2006. 25(2): p. 91-97.
39. Haynes, J.-D., K. Sakai, G. Rees, S. Gilbert, C. Frith,
and R.E. Passingham, Reading Hidden Intentions in the
Human Brain. Current Biology, 2007. 17(4): p. 323-328.
40. Hertzum, M., Images of Usability. International Journal
of Human-Computer Interaction, 2010. 26(6): p. 567-
600.
41. Hill, A.P. and C.J. Bohil, Applications of Optical
Neuroimaging in Usability Research. Ergonomics in
design : the magazine of human factors applications,
2016. 24(2): p. 4-9.
42. Hirshfield, L.M., E.T. Solovey, A. Girouard, J.
Kebinger, R.J.K. Jacob, A. Sassaroli, and S. Fantini,
Brain measurement for usability testing and adaptive
interfaces: an example of uncovering syntactic workload
with functional near infrared spectroscopy, in ACM CHI
Conference on Human Factors in Computing Systems.
2009, ACM: Boston, MA, USA. p. 2185-2194.
43. Hornbæk, K., Current practice in measuring usability:
Challenges to usability studies and research.
International Journal of Human-Computer Studies, 2006.
64(2): p. 79-102.
44. Huettel, S.A., A.W. Song, and G. McCarthy, Functional
Magnetic Resonance Imaging. 2009: Sinauer Associates,
Incorporated.
45. Jacobs, R.H.A.H., R. Renken, and F.W. Cornelissen,
Neural Correlates of Visual Aesthetics – Beauty as the
Coalescence of Stimulus and Internal State. PLoS ONE,
2012. 7(2): p. e31248.
46. Jacobsen, T., R.I. Schubotz, L. Hofel, and D.Y. Cramon,
Brain correlates of aesthetic judgment of beauty.
Neuroimage, 2006. 29(1): p. 276-85.
47. Kampe, K.K., C.D. Frith, R.J. Dolan, and U. Frith,
Reward value of attractiveness and gaze. Nature, 2001.
413(6856): p. 589.
48. Karapanos, E., User Experience Over Time, in Modeling
Users' Experiences with Interactive Systems. 2013,
Springer Berlin Heidelberg: Berlin, Heidelberg. p. 57-
83.
49. Kawabata, H. and S. Zeki, Neural correlates of beauty. J
Neurophysiol, 2004. 91(4): p. 1699-705.
50. Kim, H., R. Adolphs, J.P. O'Doherty, and S. Shimojo,
Temporal isolation of neural processes underlying face
preference decisions. PNAS, 2007. 104(46): p. 18253-
18258.
51. Kühn, S. and J. Gallinat, The neural correlates of
subjective pleasantness. NeuroImage, 2012. 61(1): p.
289-294.
52. Kurth, R., K. Villringer, G. Curio, K.J. Wolf, T. Krause,
J. Repenthin, J. Schwiemann, M. Deuchert, and A.
Villringer, fMRI shows multiple somatotopic digit
representations in human primary somatosensory cortex.
Neuroreport, 2000. 11(7): p. 1487-91.
53. Landauer, T.K., The Trouble with Computers:
Usefulness, Usability, and Productivity. 1996: MIT
Press.
54. Laugwitz, B., T. Held, and M. Schrepp, Construction
and Evaluation of a User Experience Questionnaire, in
HCI and Usability for Education and Work, A.
Holzinger, Editor. 2008, Springer Berlin Heidelberg. p.
63-76.
55. Law, E.L.-C., V. Roto, M. Hassenzahl, A.P.O.S.
Vermeeren, and J. Kort, Understanding, scoping and
defining user experience: a survey approach, in CHI.
2009, ACM: Boston, MA, USA. p. 719-728.
56. Lewis, J.R., Usability: Lessons Learned … and Yet to
Be Learned. International Journal of Human-Computer
Interaction, 2014. 30(9): p. 663-684.
57. Lindgaard, G. and C. Dudek, What is this evasive beast
we call user satisfaction? Interacting with Computers,
2003. 15(3): p. 429-452.
58. Lindgaard, G., C. Dudek, D. Sen, L. Sumegi, and P.
Noonan, An exploration of relations between visual
appeal, trustworthiness and perceived usability of
homepages. ACM Trans. Comput.-Hum. Interact., 2011.
18(1): p. 1-30.
59. Lindgaard, G., G. Fernandes, C. Dudek, and J. Brown,
Attention web designers: you have 50 milliseconds to
make a good first impression!
60. Lukanov, K., H.A. Maior, and M.L. Wilson, Using
fNIRS in Usability Testing: Understanding the Effect of
Web Form Layout on Mental Workload, in Proceedings
of the 2016 CHI Conference on Human Factors in
Computing Systems. 2016, ACM: Santa Clara,
California, USA. p. 4011-4016.
61. McCandliss, B.D., L. Cohen, and S. Dehaene, The visual
word form area: expertise for reading in the fusiform
gyrus. Trends in Cognitive Sciences, 2003. 7(7): p. 293-
299.
62. Mecklinger, A., C. Gruenewald, N. Weiskopf, and C.F.
Doeller, Motor affordance and its role for visual working
memory: evidence from fMRI studies. Exp Psychol,
2004. 51(4): p. 258-69.
63. Miller, E.K., D.J. Freedman, and J.D. Wallis, The
prefrontal cortex: categories, concepts and cognition.
Philosophical Transactions of the Royal Society B:
Biological Sciences, 2002. 357(1424): p. 1123-1136.
64. Molenberghs, P., R. Cunnington, and J.B. Mattingley, Is
the mirror neuron system involved in imitation? A short
review and meta-analysis. Neuroscience &
Biobehavioral Reviews, 2009. 33(7): p. 975-980.
65. Moll, J., P.J. Eslinger, and R. Oliveira-Souza,
Frontopolar and anterior temporal cortex activation in a
moral judgment task: preliminary functional MRI results
in normal subjects. Arq Neuropsiquiatr, 2001. 59(3-b):
p. 657-64.
66. Monti, M.M., L.M. Parsons, and D.N. Osherson, The
boundaries of language and thought in deductive
inference. PNAS, 2009. 106(30): p. 12554-12559.
67. Moshagen, M. and M.T. Thielsch, Facets of visual
aesthetics. International Journal of Human-Computer
Studies, 2010. 68(10): p. 689-709.
68. Nielsen, J., Usability Engineering. 1994: AP
Professional.
69. Paulus, M.P. and L.R. Frank, Ventromedial prefrontal
cortex activation is critical for preference judgments.
Neuroreport, 2003. 14(10): p. 1311-5.
70. Pedersen, E.W., S. Subramanian, and K. Hornbæk. Is
my Phone Alive? A Large-Scale Study of Shape Change
in Handheld Devices Using Videos. CHI. 2014. ACM.
71. Peirce, C.S. and J. Buchler, Philosophical Writings of
Peirce. 2012: Dover Publications.
72. Phillips, M.L., L. Williams, C. Senior, E.T. Bullmore,
M.J. Brammer, C. Andrew, S.C.R. Williams, and A.S.
David, A differential neural response to threatening and
non-threatening negative facial expressions in paranoid
and non-paranoid schizophrenics. Psychiatry Research:
Neuroimaging, 1999. 92(1): p. 11-31.
73. Pine, D.S., J. Grun, E.A. Maguire, N. Burgess, E.
Zarahn, V. Koda, A. Fyer, P.R. Szeszko, and R.M.
Bilder, Neurodevelopmental aspects of spatial
navigation: a virtual reality fMRI study. Neuroimage,
2002. 15(2): p. 396-406.
74. Pochon, J.B., R. Levy, P. Fossati, S. Lehericy, J.B.
Poline, B. Pillon, D. Le Bihan, and B. Dubois, The
neural system that bridges reward and cognition in
humans: An fMRI study. PNAS, 2002. 99(8): p. 5669-
5674.
75. Poldrack, R.A., Can cognitive processes be inferred
from neuroimaging data? Trends Cogn Sci, 2006. 10(2):
p. 59-63.
76. Pólya, G., Mathematics and Plausible Reasoning:
Induction and analogy in mathematics. 1990: Princeton
University Press.
77. Purves, D., Neuroscience. 2012: Sinauer Associates.
78. Rizzolatti, G. and L. Craighero, The mirror-neuron
system. Annu Rev Neurosci, 2004. 27: p. 169-92.
79. Sauro, J. and J.R. Lewis, Correlations among
prototypical usability metrics: evidence for the construct
of usability, in CHI. 2009, ACM: Boston, MA, USA. p.
1609-1618.
80. Sjölie, D., K. Bodin, E. Elgh, J. Eriksson, L.-E. Janlert,
and L. Nyberg, Effects of interactivity and 3D-motion
on mental rotation brain activity in an immersive virtual
environment, in CHI. 2010, ACM: Atlanta, Georgia,
USA. p. 869-878.
81. Tractinsky, N., A. Cokhavi, M. Kirschenbaum, and T.
Sharfi, Evaluating the consistency of immediate
aesthetic perceptions of web pages. IJHCS, 2006.
64(11): p. 1071-1083.
82. Tuch, A.N., S.P. Roth, K. Hornbæk, K. Opwis, and J.A.
Bargas-Avila, Is beautiful really usable? Toward
understanding the relation between usability, aesthetics,
and affect in HCI. Computers in Human Behavior, 2012.
28(5): p. 1596-1607.
83. Vartanian, O. and V. Goel, Neuroanatomical correlates
of aesthetic preference for paintings. Neuroreport, 2004.
15(5): p. 893-7.
84. Vartanian, O., G. Navarrete, A. Chatterjee, L.B. Fich, H.
Leder, C. Modroño, M. Nadal, N. Rostrup, and M. Skov,
Impact of contour on aesthetic judgments and approach-
avoidance decisions in architecture. PNAS, 2013.
110(Supplement 2): p. 10446-10453.
85. Vi, C., K. Takashima, H. Yokoyama, G. Liu, Y. Itoh, S.
Subramanian, and Y. Kitamura, D-FLIP: Dynamic and
Flexible Interactive PhotoShow, in Advances in
Computer Entertainment, D. Reidsma, H. Katayose, and
A. Nijholt, Editors. 2013, Springer. p. 415-427.
86. Yamamoto, S., I.E. Monosov, M. Yasuda, and O.
Hikosaka, What and where information in the caudate
tail guides saccades to visual objects. J Neurosci, 2012.
32(32): p. 11005-16.
