Introduction to Cognitive Modeling: Architectures and Approaches

Q550: Models in Cognitive Science

Lecture 2

Review Ø  We have a foggy idea what we’re supposed to be doing: no

text, no universal definition of a model, no accepted practice for designing and evaluating models

Ø  BUT: Over 85% of new pubs involve cognitive modeling

Ø  Features of a Cognitive Model: 1.  Goal is to explain basic cognitive processes 2.  Described in formal languages 3.  Derived from basic principles of cognition

Ø  Advantages of Cognitive Models: 1.  Produce logically valid predictions 2.  Capable of making precise quantitative predictions 3.  Capable of generalizing

“Computational modeling requires the researcher to be explicit about a theory in a way that a verbal theory does not” …It removes the “free wiggle room” that comes with all verbal theories

- Murphy (2011)

“Over the Christmas Holiday, Alan Newell and I invented a thinking machine”

- Herb Simon, January 1956

First Caveat: All models are wrong “Regardless of their form or function, or the area in which they are used, it is safe to say that these models all have one thing in common: They are all wrong” -MacCallum (2003)

“Modeling … has often been misunderstood and incorrectly characterized, because scientists have a natural tendency to act as if models are either right or wrong, engaging in a kind of scientific combat to establish truth. Of course, none of our models are ever correct, even when restricted to the simplest and most controlled experimental settings.” -Shiffrin (2010)

Maps as models

A “Model” of a system:

A “Model” of a system: Components:

•  Formal description

•  Structure and function

•  Direction (cause/effect)

•  Assumptions

•  Parameters

Goals:

1.  Description/Prediction

2.  Process discrimination

3.  Explanation

A “Model” of a system:

A “Model” of a system:

Another (weird) pattern of data:

Nosofsky, R. M. (1991). Tests of an exemplar model for relating perceptual classification and recognition memory. Journal of Experimental Psychology: HPP, 17, 3-27.

Another (weird) pattern of data:

Nosofsky, R. M. (1991). Tests of an exemplar model for relating perceptual classification and recognition memory. Journal of Experimental Psychology: HPP, 17, 3-27.

What is Cognitive Modeling?

“Cognitive science is concerned with understanding the processes that the brain uses to accomplish complex tasks including perceiving, learning, remembering, thinking, predicting, inference, problem solving, decision making, planning, and moving around the environment. The goal of a cognitive model is to scientifically explain one or more of these basic cognitive processes, or explain how these processes interact.”

--Busemeyer & Diederich (2010)

How do we differentiate a cognitive model from a conceptual or statistical model?

Cognitive Processes �  Cognition: “The process of thought” …?

�  cf. a physical process (e.g., gravity), we want to model a cognitive process (e.g., forgetting)

�  Traditionally, we are interested in the how, not where

�  However, the where constrains the how: �  Structure and function: Inseparable (?); structure constrains what can be computed

Modeling an “intelligent” system: What is going on in Homer’s head?

y = xi⋅ (−1)i−1

i=1

N

∑ −1

(Digit1 - Digit2 + Digit3 - Digit4 …) - 1

“If the input is positive, Homer adds the odd digits (read left to right), and subtracts even digits, and then subtracts 1 at the end to be safe. So, given the input abcd, Homer would output a-b+c-d-1. If the input is negative, Homer subtracts the odd digits, and adds the even digits, subtracting 1 at the end (to be safe). So, given the input abcd, Homer would output -a+b-c+d-1.”

for i = 1, len(string) do resp = resp + string(i) * (2*(mod(i,2)-1) end resp = resp - 1

If ± ...edcba is an integer such that a is the digit in the ones place, b the digit in the tens place, c the digit in the hundreds place, etc., then ± ...edcba returns -1 ± (a - b + c - d + e - ....) if there are an odd number of digits, and returns -1 ± (- (a - b + c - d + e - ....)) if there are an even number of digits.

Features of a Cognitive Model:

•  VCTs are broadly stated theoretical assumptions

•  Cognitive models differ in that they are formally stated

•  E.g., Craik and Lockhart’s LOP hypothesis provides a conceptual framework for memory, whereas SAM or TODAM, being formal, are examples of cognitive models

•  We can convert a conceptual framework into a cognitive model by formalizing it (recasting the verbal statements into mathematical or computer language)

1)  Goal is to scientifically explain basic cognitive processes

2)  Described in formal (mathematical or computer) languages

Features of a Cognitive Model:

•  Statistical models are applicable to data from any field, as long as those data meet the statistical assumptions (normality, etc.)

•  These assumptions are not derived from any principles of cognition, and may be inconsistent with known facts of cognition

Ø  E.g., statistical assumptions are inconsistent w/ RT dists

Ø  Cognitive models of response time accommodate this

•  Statistical vs. explanatory models (CSLR/SVM car)

•  Parameters should have a meaningful cognitive interpretation

1)  Goal is to scientifically explain basic cognitive processes

2)  Described in formal (mathematical or computer) languages

3)  Derived from basic principles of cognition

Advantages of Cognitive Models:

•  This is not true of conclusions based on intuitively-based verbal reasoning

•  E.g., early categorization research argued against exemplar models b/c transfer to a prototype was better than any of the experienced exemplars

•  But once the exemplar model was formalized, it naturally produces this effect

•  Reasoning from a conceptual framework can lead to incorrect conclusions (Goat game)

1)  Guaranteed to produce logically valid predictions

Advantages of Cognitive Models

•  Even if a model predicts a correct qualitative (ordinal) effect, its predictions may be an order of magnitude off the mark

•  Formal models allow us to evaluate models both qualitatively and quantitatively in their correspondence to empirical data

•  Gallileo’s balls

1)  Guaranteed to produce logically valid predictions

2)  Capable of making precise quantitative predictions

Advantages of Cognitive Models

•  Both cognitive models and statistical models (empirical curve fitting) are capable of generating quantitative predictions

•  But cognitive models should also generalize to new paradigms, and should describe the process that produces the output, rather than just describing the output itself

•  Newell’s (1981) power law vs. Logan’s (1988) instance theory Ø  Logan’s theory produces and explains the power law, and can

be used to make new predictions: how variance of RT dist changes w/ practice and how accuracy changes w/ practice

1)  Guaranteed to produce logically valid predictions

2)  Capable of making precise quantitative predictions

3)  Capable of generalizing

Classifying Models: Lewandowsky & Farrell (2011)

1.   Data Description (input-output function)

2.   Process Identification/Characterization (peek inside black box; what processes are producing input-output)

3.   Process Explanation (explain how black box is producing output from input)

1. Descriptive Models Ø  Statistical models are primarily descriptive/predictive

Ø  Primary goal is a reduced description of the data at hand

Ø  Practice Functions: •  When learning a new skill, are time savings from practice

following a power law of practice?

•  Note: Choice of correct descriptive model can have important implications about the psychological nature of a phenomenon (here, learning)

Heathcote,  Brown,  &  Mewhort  (2000):  The  power  law  repealed:  The  case  for  an  exponen<al  law  of  prac<ce.  Psychonomic  Bulle/n  &  Review,  7,  185-­‐207.    

RT = t−β

RT = e−αt

RT = t−α

RT = e−βt “One attribute of descriptive models is that they are

explicitly devoid of psychological content; for example, although the existence of an exponential practice function constrains possible learning mechanisms, the function itself has no psychological content. It is merely concerned with describing the data.”

- Lewandowsky & Farrell (2011)

2. Characterization Models Ø  Ability to identify cognitive constructs producing a phenomenon

Ø  They postulate and measure processes, but don’t offer a detailed explanation of how the process works

Ø  “Unlike descriptive models, explanatory power rests on hypothetical constructs within the mind rather than within the data to be explained” (L & F, 2011).

Ø  However, these models do not go beyond identification of these processes—they are agnostic about explaining how the processes work (e.g., LOP)

Ø  Example: Multinomial Processing Trees (MPTs): •  E.g., Batchelder & Riefer, 1999

3. Explanatory Models Ø  Most detailed view inside the black box

Ø  Also posit hypothetical cognitive constructs, but with a detailed explanation of how these processes might work

Ø  Why bother with characterization; why not just explain everything?

Ø  Not always possible; not always useful

Examples with Nosofsky’s (1986) GCM and Hintzman’s (1986) MINERVA 2

Steps of Cognitive Modeling: 1)  Conceptual theory formal description

Steps of Cognitive Modeling:

•  Conceptual theory is often missing important details, so we have to make additional detailed assumptions in order to complete the model and to generate quantitative predictions

•  E.g., assumptions about features to represent stimuli

•  We try to minimize the number of ad hoc assumptions, but this step is often unavoidable

1)  Conceptual theory formal description

2)  Ad hoc assumptions to complete formal description

Steps of Cognitive Modeling:

•  Models almost always contain coefficients that are initially unknown, and these values need to be estimated from some of the observed data

•  E.g., the importance weight assigned to each feature is a free parameter that is estimated from the choice response data

•  We try to minimize the number of model parameters, but this is usually a necessary and important step

1)  Conceptual theory formal description

2)  Ad hoc assumptions to complete formal description

3)  Parameter estimation

Steps of Cognitive Modeling:

•  Qualitative: Ordinal and parameter free (models make these predictions for any value of the free parameters)

•  Quantitative: Magnitude of correspondence to data

•  Cognitive models can be compared to each other quantitatively, or we can use a base and saturated model

1)  Conceptual theory formal description

2)  Ad hoc assumptions to complete formal description

3)  Parameter estimation

4)  Compare predictions to empirical data

Steps of Cognitive Modeling:

•  Experiments are designed based on models, and models are constrained and modified/extended based on new data

•  This should produce an evolution of models that improve and become more powerful over time as the science in a field progresses

1)  Conceptual theory formal description

2)  Ad hoc assumptions to complete formal description

3)  Parameter estimation

4)  Compare predictions to empirical data

5)  Iterate to constrain

Evolution of Math Modeling Estes, W. K. (1975). Some targets for mathematical psychology. Journal of Mathematical Psychology, 12, 263-282.

In looking back on math modeling, Estes notes:

•  We lack cumulative progress (nobody builds on models, they just build new ones)

•  We’re modeling tasks, not general processes

• Math psyc isn’t helpful to any applied problems

•  We need to start archiving data

•  We can’t just fit a descriptive model, we need an explanatory and generalizable model

Evolution of Math Modeling •  We need agreed upon methods for model comparison and fit

•  We fit models to highly controlled experiments, and we may be unknowingly hold constant factors which would have a major effect if allowed to vary (Ecological validity vs. control --> Large-scale models)

•  These computer things can’t be trusted…

Why computers are dangerous Computers allow us to create theory “…almost completely bypassing both the opportunities for ingenuity and the quantities of blood, sweat, and tears that used to go into this enterprise.” (p. 268)

This came true, and perhaps a bigger problem now: You don’t need wisdom to whip up a model. (Youth creating theory)

“Is mathematical psychology in danger of becoming obsolescent before reaching maturity?” (p. 279)

If you were to write a “Targets for cognitive models” paper today, what wishes would you come up with?

Lewandowsky, S., & Heit, E. (2006). Some targets for memory models. Journal of Memory and Language, 55, 441-446.

An argument: �  Instances vs. sets and averages:

•  Ancient theory of eidola: things fly into your head

•  Plato/Aristotle: Essential elements [+wings] [+beak] [+flies]

�  Instances vs. Protypes: •  Rosch/Lakoff: Prototypes, basic levels have special processing advantage, cognitive economy

•  Brooksian: Inflexible to only store prototype

�  Exemplar-based vs. Prototype models: •  Exemplars: Hintzman, Nosofsky,

•  Prototypes: Reed

"   Rosch (1973, 1975): Typical members of a semantic category can be processed more efficiently than atypical ones

Typicality Effects

High Med Low

0.52

0.54

0.56

0.58

0.60

0.62

0.64

1.10

1.15

1.20

1.25

1.30

High Med LowHigh Med Low

1000

1050

1100

1150

1200

Euclid

ean D

ista

nce

RT

(m

s)

Typicality

Rosch Prototype Label

(Lets use novel stimuli so we have control)

Internal Representation

Tasio

Tasio Radok

Radok

Galli Galli

Samar

Internal Representation “Samar”

Prototype model vs. exemplar model

Formalizing a model:

Representation: Random vectors for Greebles

Process:

•  Prototype averages within each category

•  Recency?

•  Exemplar: Instances are each stored with pattern label

•  Similarity is Euclidean distance in both:

•  There is randomness in the response process

1)  Conceptual Theory --> Formal Model

2)  Ad-hoc assumptions:

€

dij = (xim − x jm

m∈M

∑ )

Formalizing a model:

Fixed:

•  Task: num_per_cat, trials, etc.

•  Cognitive: num_features, n_flip, etc.

Free:

•  Recency weighting, response sensitivity, distance scaling, attention

When we’re done, the models make precise predictions, and the only difference between models should be the theoretical debate

1)  Conceptual Theory --> Formal Model

2)  Ad-hoc assumptions

3)  Parameters:

More Background Sun, R. (2008). Introduction to computational modeling. Cambridge Handbook of Computational Psychology. Cambridge University Press.

Estes, W. K. (2002). Traps in the route to a model of memory and decision. Psychonomic Bulletin & Review, 9, 3-25.

The OSU Cognitive Modeling Repository

Next Class

An existence proof: Hintzman, D. L. (1986). 'Schema abstraction' in a multiple-trace memory model. Psychological Review, 93, 411-428.

Read some history and a modern manifesto:

McClelland, J. L. (2009). The place of modeling in cognitive science. Topics in Cognitive Science, 1, 11-38.

Addyman, C., & French, R. M. (2012). Computational modeling in cognitive science: A manifesto for change. Topics in Cognitive Science, 4, 332-341.