Given the similarly of vision and visual imagery, does this suggest that imaging involves a mental picture?

Reasoning from the involvement of the visual Reasoning from the involvement of the visual system in mental imagery to the conclusion system in mental imagery to the conclusion that the format of the image must be pictorial, that the format of the image must be pictorial, is based on a thoroughly discredited theory of is based on a thoroughly discredited theory of vision (the picture theory of vision).vision (the picture theory of vision). • There are serious reasons to doubt that There are serious reasons to doubt that

vision vision involves the construction of a involves the construction of a pictorial displaypictorial display

Conscious experience and the picture-theoryThe picture theory was meant to explain why our perceptual experience is panoramic and stable while the visual inputs are partial and constantly changing. This assumes that the content of experience is represented.

But the picture theory of vision has been thoroughly discredited: There is no rich panoramic display in vision (e.g., see change blindness, superposition studies, …)

Vision appears to fill in missing parts of a familiar pattern

Find three of these …

In here….

But superposition of images has been discredited

O'Regan & Lévy-Schoen, 1983

We encode far less visual information than we generally believe we do – far from what any picture would hold

Harborside

Airplane

Signature properties of vision do not apply to off-retinal information

Signature properties of vision do not apply to off-retinal information

Off-retinal information uses memory

Effectiveness of anorthoscopic presentation depends on memory load

There are many questions about what goes on when we have the experience of “seeing with the mind’s eye”

Is mental imagery a special form of thought? If so, in what way is it special? Are mental images sensory-based and modality-specific? Are mental images like pictures? In what respect? Are images different from other forms of thought? Do

they resemble what they represent? Does mental imagery use the visual system? If so, what does

that tell us about the format of images? Is there neurophysiological evidence for a pictorial “display”

in visual cortex? What if a display were found in human visual cortex?

A central methodological point:The difference between appeals to properties of the architecture or to properties of what is represented

Suppose we observe some robust behavioral regularity. Suppose we observe some robust behavioral regularity. What might it tell us about the nature of the system that What might it tell us about the nature of the system that exhibits the regularity?exhibits the regularity?

e.g., in the case of human behavior, what does it e.g., in the case of human behavior, what does it tell us about the cognitive architecture or the format tell us about the cognitive architecture or the format of representations?of representations?

An illustrative example: Behavior of a mystery box

What does this behavior pattern tell us about the nature of the box?

The moral of this example: Regularities in behavior may be due to one of two very different causes:

1.1. The inherent nature of the system The inherent nature of the system (to its structure or wiring), or(to its structure or wiring), or

2.2. The nature of what the system The nature of what the system represents (what it “knows”).represents (what it “knows”).

The major difference between picture-theorists and the rest of us (me) is in how we answer the following question:

1.1. Do experiments on mental imagery (such as those I will Do experiments on mental imagery (such as those I will review) tell us anything about the format of images or review) tell us anything about the format of images or about the properties of a special imagery architecture? about the properties of a special imagery architecture? Or …Or …

2.2. Do the experiments tell us about the knowledge that Do the experiments tell us about the knowledge that people have about how things would look if they were people have about how things would look if they were actually to actually to seesee them (together with some common them (together with some common psychophysical skills)?psychophysical skills)? While these are the main alternatives, there are also While these are the main alternatives, there are also

other reasons why experiments come out as they do.other reasons why experiments come out as they do.

The “imagery debate”

Imagine various events unfolding before your “mind’s eye” –

Imagine a bicyclist racing up a hill. Down a hill?Imagine a bicyclist racing up a hill. Down a hill? Imagine turning a large heavy wheel. A light wheel.Imagine turning a large heavy wheel. A light wheel. Imagine a baseball being hit. What shape trajectory Imagine a baseball being hit. What shape trajectory

does it trace out? Where would you run to catch it?does it trace out? Where would you run to catch it? Imagine a coin dropping and whirling on its edge as Imagine a coin dropping and whirling on its edge as

it eventually settles. How does it behave?it eventually settles. How does it behave? Imagine a heavy ball (a shot-put) being dropped at Imagine a heavy ball (a shot-put) being dropped at

the same time as a light ball (a tennis ball). Indicate the same time as a light ball (a tennis ball). Indicate when they hit the floor. Repeat for different heights.when they hit the floor. Repeat for different heights.

Examples to probe your intuition

Why did things unfold the way they did in your imagination?

Was it because of the Was it because of the formatformat of your image or your of your image or your cognitive cognitive architecturearchitecture? Or because of what you know?? Or because of what you know? Did it reveal a Did it reveal a capacitycapacity of mind? of mind? Or was it because you Or was it because you mademade it it do what it did? do what it did?

Could you make your image have any properties you Could you make your image have any properties you choose? Or behave in choose? Or behave in any wayany way you want? Why not? you want? Why not? How about imagining an object from all directions at once, or How about imagining an object from all directions at once, or

from no particular direction?from no particular direction? How about imagining a 4-dimensional object?How about imagining a 4-dimensional object? Can you imagine a printed letter which is neither upper nor Can you imagine a printed letter which is neither upper nor

lower case? A triangle that is not a particular type?lower case? A triangle that is not a particular type?

More examples of the behavior of images: Do they tell us about tacit knowledge or about the nature of the architecture or format

Color mixing Color mixing Conservation of volumeConservation of volume

Or, more relevant to the present discussion:Or, more relevant to the present discussion: The effect of image sizeThe effect of image size Scanning mental imagesScanning mental images

Studies of mental scanningDoes it show that images have metrical space?

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Relative distance on image

Late

ncy (

secs)

scan image

imagine lights

show direction

(Pylyshyn & Bannon. See Pylyshyn, 1981)

Conclusion: The image scanning effect is Cognitively Penetrable i.e., it depends on goals and beliefs, or on Tacit Knowledge.

Another methodological note What is tacit knowledge?What is tacit knowledge?

A central construct in cognitive science: Cognitive A central construct in cognitive science: Cognitive phenomena turn on goals and beliefsphenomena turn on goals and beliefs

Tacit knowledge need not be consciously accessible but Tacit knowledge need not be consciously accessible but may be inferred indirectlymay be inferred indirectly

Access to tacit knowledge is context-dependentAccess to tacit knowledge is context-dependent Tacit knowledge of what?Tacit knowledge of what?

It does not refer to knowledge of what the experimenter wants nor It does not refer to knowledge of what the experimenter wants nor the expectation of how the experiments will turn out (cf Dennis). It the expectation of how the experiments will turn out (cf Dennis). It refers to knowledge of what it would be like to refers to knowledge of what it would be like to seesee the relevant the relevant situation as it unfolds.situation as it unfolds.

Cognitive penetrabilityCognitive penetrability as a litmus as a litmus

Major Question 2: Does visual imagery use the visual system?

It depends on what you mean by “visual” and by “use”It depends on what you mean by “visual” and by “use” visionvision can involve all of cognition can involve all of cognition use visionuse vision can mean actually can mean actually perceiveperceive the image the image The reason this question is of interest to picture theorists is The reason this question is of interest to picture theorists is

that a positive answer is thought to entail that images are that a positive answer is thought to entail that images are picture-like.picture-like.

Let’s assume that “the visual system” is “active” in some Let’s assume that “the visual system” is “active” in some way during episodes of mental imageryway during episodes of mental imagery . .

What follows from that?What follows from that? Why should we think that vision may be involved?Why should we think that vision may be involved?

Note that the main support for the assumption that vision is involved Note that the main support for the assumption that vision is involved comes from neuroscience – to be considered later.comes from neuroscience – to be considered later.

Reasons for thinking that images are interpreted through the visual system

Similar phenomenology of imagining & seeingSimilar phenomenology of imagining & seeing This is the absolutely central reason which This is the absolutely central reason which

overshadows all othersovershadows all others Superposition & interference studiesSuperposition & interference studies Visual illusions with projected imagesVisual illusions with projected images

The ubiquitous role of attentionThe ubiquitous role of attention Reperceiving and novel reconstrualsReperceiving and novel reconstruals

A large but very problematic literature A large but very problematic literature Chambers, Finke, Peterson, SlezakChambers, Finke, Peterson, Slezak

More persuasive demonstrations

Images constructed from descriptionsImages constructed from descriptions The umbrella example(s)The umbrella example(s) The two-parallelogram exampleThe two-parallelogram example

Amodal completionAmodal completion Reconstruals: Peterson Reconstruals: Peterson vsvs Slezak Slezak

Imagine two parallelograms

Imagine two parallelograms

Imagine two parallelograms

Amodal completion by imagery?

Amodal completion by imagery?

Vision is involved when images are superimposed onto visual displays

Many experiments show that when you project Many experiments show that when you project an image onto a display the image acts very an image onto a display the image acts very much like a superimposed displaymuch like a superimposed display• Shepard & Podgorny, Hayes, …Shepard & Podgorny, Hayes, …• Interference effects (Brooks)Interference effects (Brooks)• Mental scanningMental scanning• Interaction with the motor system (Finke)Interaction with the motor system (Finke)

Can images be visually reinterpreted? There have been many claims that people can visually There have been many claims that people can visually

reinterpret images reinterpret images ((but remember the parallelogram example!but remember the parallelogram example!)) These have all been cases where one could easily figure These have all been cases where one could easily figure

out what the combined image would look like without out what the combined image would look like without actually seeing it (e.g., the J – D superposition).actually seeing it (e.g., the J – D superposition).

Pederson’s careful examination of visual “reconstruals” Pederson’s careful examination of visual “reconstruals” showed (contrary to her own conclusion) that images showed (contrary to her own conclusion) that images are never ambiguous (no Necker cube or figure-ground are never ambiguous (no Necker cube or figure-ground reversals) and when new construals were achieved from reversals) and when new construals were achieved from images they were quite different from the ones achieved images they were quite different from the ones achieved in vision (more variable, more guessing from partial in vision (more variable, more guessing from partial cues, etc).cues, etc).

The best evidence comes from a philosopher, Peter The best evidence comes from a philosopher, Peter Slezak!Slezak!

Slezak figures

Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.

Slezak figures rotated 90o

And what if visual imagery did involve the “visual system”?

Even if there was credible evidence that vision Even if there was credible evidence that vision is involved in visual imagery, it would tell us is involved in visual imagery, it would tell us nothing about the nothing about the formatformat of imagery because, of imagery because,

There is strong evidence that There is strong evidence that visualvisual representations are not pictorial either!representations are not pictorial either!

The picture theory of visionis a non-starter, even for cats

(Cartoon by Kliban)

A more plausible theory of vision(even for cats)